PARQUET-306: Add row group alignment

[rdblue]

This adds AlignmentStrategy to the ParquetFileWriter that can alter the position of row groups and recommend a target size for the next row group. There are two strategies: NoAlignment and PaddingAlignment. Padding alignment is used for HDFS and no alignment is used for all other file systems. When HDFS-3689 is available, we can add a strategy to use that.

The amount of padding is controlled by a threshold between 0 and 1 that controls the fraction of the row group size that can be padded. This is interpreted as the maximum amount of padding that is acceptable, in terms of the row group size. For example, setting this to 5% will write padding when the bytes left in a HDFS block are less than 5% of the row group size. This defaults to 0%, which prevents padding from being added and matches the current behavior. The threshold is controlled by a new OutputFormat configuration property, parquet.writer.padding-thresh.

[rdblue]

@isnotinvain could you look at this before the ParquetWriter builder? If we can get this in, then I'll have to update the builder class to set the padding threshold.

[isnotinvain]

should this just throw if !hasNonNullValue?
We already provide hasNonNullValue, shouldn't the caller inspect that before calling getMaxBytes?

[rdblue]

This gets called from within Jackson when converting to JSON, so we don't have much of a choice.

[isnotinvain]

ok SGTM

[isnotinvain]

How do we make sure that row groups are still aligning with HDFS blocks? Everything gets written to an output stream, so it seems like it would be really easy to accidentally shift everything slightly and ruin the alignment -- eg we write a magic header at the beginning of the file right? does that push everything slightly out of alignment? It only has to be slightly off for any wins to be lost right?

I'm just wondering if we're relying on being very careful about that, or if the way this works is not sensitive to that (maybe it's not because it asks the output stream for it's position, which is all that matters maybe?)

[rdblue]

How do we make sure that row groups are still aligning with HDFS blocks?

This is based off of the position that the FS OutputStream reports, which is always going to be correct. If some other thread somehow had a reference to it and wrote data, then this would see that the position had changed.

I'm also going to update this so that we don't use a percentage of the row group size to configure it. I'm going to add a max padding setting in bytes instead.

[rdblue]

Ok, changed padding thresh to maxPaddingSize. I think that's much better.

[rdblue]

I've fixed the tests and added webhdfs and viewfs to the supported block schemes. Should be good to go?

[isnotinvain]

nit, but how about setMaxRowGroupPaddingBytes

[isnotinvain]

What do you think about adding one more test, that writes a parquet file (with padding) and then reads it back (there are already lots of tests that do this, but w/o padding). I remember seeing discussion around a different issue where parquet stores an offset for the dictionary header, but never actually reads it, just assuming that the dictionary is the first page, and so we never noticed that the offset was wrong, etc. I guess I'm saying it'd be good to sanity check that parquet still reads files with padding in them w/o issue.

[rdblue]

@isnotinvain, good idea. I originally set the default padding to non-zero so all the tests tested that, which is why you don't see one. I still think we should change the default to include a reasonable amount of padding, but probably not in this PR.

[rdblue]

I added the test, but I didn't change the method names.

[rdblue]

Rebased on top of #221 and tests are passing.

[isnotinvain]

probably want to getBytes("UTF-8") right?

[isnotinvain]

+1, just one comment about character encodings in the tests