Skip to content

Support merge manifests on writes (MergeAppend) #363

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fokko merged 22 commits into from
Jul 10, 2024
Merged

Support merge manifests on writes (MergeAppend) #363

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6803eba
add ListPacker + tests
HonahX Jun 3, 2024
f0fc260
add merge append
HonahX Jun 3, 2024
cbb8cec
add merge_append
HonahX Jun 3, 2024
bf63c03
fix snapshot inheritance
HonahX Jun 3, 2024
9dd69af
test manifest file and entries
HonahX Jun 4, 2024
4921a7f
add doc
HonahX Jun 4, 2024
984ca41
fix lint
HonahX Jun 4, 2024
8510f71
change test name
HonahX Jun 4, 2024
1ec5edd
Merge branch 'main' into manifest_compaction
HonahX Jun 10, 2024
a7da318
address review comments
HonahX Jun 10, 2024
c4feda5
rename _MergingSnapshotProducer to _SnapshotProducer
HonahX Jun 10, 2024
7e6e1d4
Merge branch 'main' into manifest_compaction
HonahX Jun 24, 2024
123f5d3
Merge branch 'main' into manifest_compaction
HonahX Jul 1, 2024
9777e9b
fix a serious bug
HonahX Jul 1, 2024
3393757
Merge branch 'main' into manifest_compaction
HonahX Jul 2, 2024
66dddbe
update the doc
HonahX Jul 2, 2024
aff1bea
remove merge_append as public API
HonahX Jul 2, 2024
7625857
make default to false
HonahX Jul 2, 2024
3e3a1b4
add test description
HonahX Jul 2, 2024
914d6ef
Merge branch 'main' into manifest_compaction
HonahX Jul 10, 2024
71a5fe0
fix merge conflict
HonahX Jul 10, 2024
c7e4095
fix snapshot_id issue
HonahX Jul 10, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
make default to false
  • Loading branch information
@HonahX
HonahX committed Jul 2, 2024
commit 7625857072a5fc0c1e58ad3422b26ec437a6dd33
2 changes: 1 addition & 1 deletion pyiceberg/table/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,7 +254,7 @@ class TableProperties:
MANIFEST_MIN_MERGE_COUNT_DEFAULT = 100

MANIFEST_MERGE_ENABLED = "commit.manifest-merge.enabled"
MANIFEST_MERGE_ENABLED_DEFAULT = True
MANIFEST_MERGE_ENABLED_DEFAULT = False


class PropertyUtil:
Expand Down
11 changes: 8 additions & 3 deletions tests/integration/test_writes/test_writes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -933,19 +933,24 @@ def test_merge_manifests(session_catalog: Catalog, arrow_table_with_null: pa.Tab
tbl_a = _create_table(
session_catalog,
"default.merge_manifest_a",
{"commit.manifest.min-count-to-merge": "1", "format-version": format_version},
{"commit.manifest-merge.enabled": "true", "commit.manifest.min-count-to-merge": "1", "format-version": format_version},
[],
)
tbl_b = _create_table(
session_catalog,
"default.merge_manifest_b",
{"commit.manifest.min-count-to-merge": "1", "commit.manifest.target-size-bytes": "1", "format-version": format_version},
{
"commit.manifest-merge.enabled": "true",
"commit.manifest.min-count-to-merge": "1",
"commit.manifest.target-size-bytes": "1",
"format-version": format_version,
},
[],
)
tbl_c = _create_table(
session_catalog,
"default.merge_manifest_c",
{"commit.manifest-merge.enabled": "false", "format-version": format_version},
{"commit.manifest.min-count-to-merge": "1", "format-version": format_version},
[],
)

Expand Down