Skip to content

PyArrow: Don't enforce the schema #902

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fokko merged 14 commits into from
Jul 11, 2024
Merged

PyArrow: Don't enforce the schema #902

Changes from 2 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
361d066
PyArrow: Don't enforce the schema
Fokko Jul 8, 2024
a0c0c57
WIP
Fokko Jul 8, 2024
4ca513b
Reuse Table schema
Fokko Jul 10, 2024
064ed0e
Merge branch 'main' of github.com:apache/iceberg-python into fd-remov…
Fokko Jul 10, 2024
ee293a1
Make linter happy
Fokko Jul 10, 2024
d2a0b36
Squash some bugs
Fokko Jul 10, 2024
3e86782
Thanks Sung!
Fokko Jul 10, 2024
8d1ed75
Moar code moar bugs
Fokko Jul 10, 2024
0fcf93c
Merge branch 'fd-remove-pushing-down-schema' of github.com:Fokko/iceb…
Fokko Jul 10, 2024
ab0db07
Remove the variables wrt file sizes
Fokko Jul 10, 2024
c4f044a
Linting
Fokko Jul 10, 2024
a760a52
Go with large ones for now
Fokko Jul 10, 2024
4464bd7
Missed one there!
Fokko Jul 11, 2024
97c5ce9
Merge branch 'main' of github.com:apache/iceberg-python into fd-remov…
Fokko Jul 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pyiceberg/io/pyarrow.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1307,7 +1307,7 @@ def _cast_if_needed(self, field: NestedField, values: pa.Array) -> pa.Array:
return values.cast(
schema_to_pyarrow(promote(file_field.field_type, field.field_type), include_field_ids=self._include_field_ids)
)
elif (target_type := schema_to_pyarrow(field.field_type, include_field_ids=True)) != values.type:
elif (target_type := schema_to_pyarrow(field.field_type, include_field_ids=self._include_field_ids)) != values.type:
# Downcasting of nanoseconds to microseconds
if (
pa.types.is_timestamp(target_type)
Expand Down