Skip to content

Support Appends with TimeTransform Partitions #784

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fokko merged 22 commits into from
May 31, 2024
Merged

Support Appends with TimeTransform Partitions #784

Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0cad231
checkpoint
sungwy May 4, 2024
96e5533
checkpoint2
sungwy May 5, 2024
ddfa9ac
todo: sort with pyarrow_transform vals
sungwy May 5, 2024
1a5327a
checkpoint
sungwy May 6, 2024
e067a28
checkpoint
sungwy May 6, 2024
069f3bd
fix
sungwy May 6, 2024
615d5e3
tests
sungwy May 6, 2024
c0a0f32
more tests
sungwy May 6, 2024
6a39eda
adopt review feedback
sungwy May 7, 2024
d14e137
comment
sungwy May 8, 2024
5dd846d
checkpoint
sungwy May 4, 2024
6357193
checkpoint2
sungwy May 5, 2024
c30a57c
todo: sort with pyarrow_transform vals
sungwy May 5, 2024
541655f
checkpoint
sungwy May 6, 2024
afe83b1
checkpoint
sungwy May 6, 2024
00ca5f0
fix
sungwy May 6, 2024
511e988
tests
sungwy May 6, 2024
3b784ab
more tests
sungwy May 6, 2024
3711b1b
adopt review feedback
sungwy May 7, 2024
f16d778
comment
sungwy May 8, 2024
80d5064
Merge branch 'transform-partition-writes' of https://github.com/syun6…
sungwy May 31, 2024
9f0a92b
rebase
sungwy May 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
checkpoint
  • Loading branch information
@sungwy
sungwy committed May 31, 2024
commit afe83b177a74039218a18cd0e49a80ff0513de1c
2 changes: 1 addition & 1 deletion pyiceberg/table/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3702,7 +3702,7 @@ def _determine_partitions(spec: PartitionSpec, schema: Schema, arrow_table: pa.T
(partition_field, schema.find_field(partition_field.source_id)) for partition_field in spec.fields
]
partition_values_table = pa.table({
str(partition.field_id): partition.pyarrow_transform(field.field_type)(arrow_table[field.name])
str(partition.field_id): partition.transform.pyarrow_transform(field.field_type)(arrow_table[field.name])
for partition, field in partition_columns
})

Expand Down