Skip to content
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions parquet/src/errors.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,9 @@ pub enum ParquetError {
/// Returned when a function needs more data to complete properly. The `usize` field indicates
/// the total number of bytes required, not the number of additional bytes.
NeedMoreData(usize),
/// Returned when a function needs more data to complete properly.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the one part of this PR I am not sure about

Since the ParquetError is (now) marked as #[non_exhaustive] I don't think this is technically a breaking change. However, it would be really nice to avoid a new enum -- I will comment about this later in the PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I had this error in my original draft of ParquetMetaDataReader (3c340b7) but got talked out of it 😅 (#6392 (comment))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should have known you were right and gone with your instinct!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW I could remove ParquetError::NeedMoreDataRange if I reworked the decoder to have a proper state machine internally (rather than calling into the existing ParquetMetaDataDecoder).

That is certainly my long term plan. If we like this API I will invest some time into seeing if I can sketch out what it would look like

/// The `Range<u64>` indicates the range of bytes that are needed.
NeedMoreDataRange(std::ops::Range<u64>),
}

impl std::fmt::Display for ParquetError {
Expand All @@ -69,6 +72,9 @@ impl std::fmt::Display for ParquetError {
}
ParquetError::External(e) => write!(fmt, "External: {e}"),
ParquetError::NeedMoreData(needed) => write!(fmt, "NeedMoreData: {needed}"),
ParquetError::NeedMoreDataRange(range) => {
write!(fmt, "NeedMoreDataRange: {}..{}", range.start, range.end)
}
}
}
}
Expand Down
7 changes: 4 additions & 3 deletions parquet/src/file/metadata/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,10 @@
//! metadata into parquet files. To work with metadata directly,
//! the following APIs are available:
//!
//! * [`ParquetMetaDataReader`] for reading
//! * [`ParquetMetaDataReader`] for reading from a reader for I/O
//! * [`ParquetMetaDataPushDecoder`] for decoding from bytes without I/O
//! * [`ParquetMetaDataWriter`] for writing.
//!
//! [`ParquetMetaDataReader`]: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataReader.html
//! [`ParquetMetaDataWriter`]: https://docs.rs/parquet/latest/parquet/file/metadata/struct.ParquetMetaDataWriter.html
//!
//! # Examples
//!
Expand Down Expand Up @@ -92,6 +91,7 @@
//! * Same name, different struct
//! ```
mod memory;
mod push_decoder;
pub(crate) mod reader;
mod writer;

Expand Down Expand Up @@ -120,6 +120,7 @@ use crate::schema::types::{
};
#[cfg(feature = "encryption")]
use crate::thrift::{TCompactSliceInputProtocol, TSerializable};
pub use push_decoder::ParquetMetaDataPushDecoder;
pub use reader::{FooterTail, PageIndexPolicy, ParquetMetaDataReader};
use std::ops::Range;
use std::sync::Arc;
Expand Down
Loading
Loading