Skip to content

feat: add RollingManifestWriter #650

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
felixscherz wants to merge 11 commits into
base: main
Choose a base branch
from
Open

feat: add RollingManifestWriter #650

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
88a1955
setup scaffolding for RollingManifestWriter
felixscherz Apr 15, 2024
1599990
feat: implement RollingManifestWriter
felixscherz Apr 21, 2024
b2160f5
test: dummy test without rollover
felixscherz Apr 21, 2024
ff58a53
test: test different rollover scenarios
felixscherz Apr 23, 2024
1cc36a5
feat: extend `OutputStream`, `AvroOutputFile` and `ManifestWriter` wi…
felixscherz May 4, 2024
c7f9611
test: patch tests temporarily
felixscherz May 4, 2024
70c37ef
feat: use `tell()` instead of `__len__`
felixscherz May 4, 2024
0527072
fix: issue with unset writer
felixscherz Jul 6, 2024
7d369ee
fix: formatting
felixscherz Jul 9, 2024
d010fd4
refactor: use _close_current_writer on exit
felixscherz Jul 9, 2024
9f01e5a
refactor: expect writer to be closed when creating manifest files
felixscherz Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: use _close_current_writer on exit
  • Loading branch information
@felixscherz
felixscherz committed Aug 5, 2024
commit d010fd453c4376e0a3918ba05e73daadecc935c9
3 changes: 1 addition & 2 deletions pyiceberg/manifest.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -875,9 +875,8 @@ def __exit__(
exc_value: Optional[BaseException],
traceback: Optional[TracebackType],
) -> None:
self._close_current_writer()
self.closed = True
if self._current_writer:
self._current_writer.__exit__(exc_type, exc_value, traceback)

def _get_current_writer(self) -> ManifestWriter:
if self._should_roll_to_new_file():
Expand Down