Use native implementation of LZMA and XZ

[hasenbanck]

This uses the native port of the XZ's liblzma. The main advantage is, that the crate is native Rust and is also used by the seven-zip2 crate.

Performance should be equal to the liblzma crate, since I spend quite a bit of time to improve both decoding and encoding performance.

I only used the single threaded version of the reader/writer, since parallelization was not enabled for liblzma. lzma-rust2 has multithreaded reader/writer though.

This PR also fixed the LZMA decoding. The old implementation actually didn't worked. The test file used STORE as compression. I checked the APPNOTE.txt to make sure that the properties are properly read. The test file is also updated (created with 7zip).

I had to change the signature of the public function Decoder::new(), since the LZMA decoder needs to know the uncompressed size to limit the range decoder. The LZMA stream CAN include an end-of-stream marker (EOS), but this is not guranteed. Most files out there have an EOS, but the specification allows LZMA streams without it (so it's best to limit the LZMA decoder with the uncompressed size). Another option

I remembered that I had to defer the initialization of reading the properties when implementing PPMd, so I did this here too.

Edit: I will have a look into what breaks the RefUnwindSafe auto trait upstream.
Edit2: UnwindSafe and RefUnwindSafe breaks because we store std::io::Error sometimes, which in itself is not UnwindSafe. Have to think about thow to handle that.

[gemini-code-assist]

Summary of Changes

Hello @hasenbanck, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the project's compression capabilities by migrating from the liblzma crate to the native Rust lzma-rust2 library. This change not only resolves a critical bug in LZMA decoding, making it fully functional, but also expands XZ support to include both compression and decompression. A key architectural adjustment was made to the LZMA decoder's initialization, requiring the uncompressed size for robust stream processing.

Highlights

• Migration to lzma-rust2: Replaced the liblzma crate with lzma-rust2 for native Rust implementation of LZMA and XZ compression, which is also used by the seven-zip2 crate.
• Improved LZMA Decoding: Fixed a critical issue where LZMA decoding was not functional, including updating the test file to validate the fix with a real-world binary file.
• Enhanced XZ Support: Enabled full XZ compression and decompression capabilities, whereas previously only decompression was supported.
• API Adjustment for LZMA: Modified the Decoder::new() signature to accept uncompressed_size, which is crucial for correct LZMA stream handling, especially for streams without an explicit end-of-stream marker.
• Optimized Initialization: Implemented deferred initialization for LZMA properties reading, aligning with best practices for stream processing and improving robustness.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a significant improvement, replacing the C-based liblzma with the native Rust lzma-rust2 crate for LZMA and XZ support. This enhances safety and simplifies the build process. The changes correctly implement LZMA decoding, which was previously broken, and add support for XZ encoding. The necessary API changes, like adding uncompressed_size to the decompressor, are well-handled. My review includes a couple of suggestions for improving code clarity in the new LZMA decoding logic and fixing a minor typo in the README.

[hasenbanck]

@Pr0methean I fixed the issue with the broken UnwindSafe and RefUndindSafe and fixed the issues marked by the code review. I had to remove the test that the AI did, since it doesn't apply to lzma-rust2. The semver check action seems incorrect in the assasment of the missing default feature, since lzma provides the same functionality now.

As a note: #408 seems dangerous, since it doesn't guard against LZMA files that don't have an EOS and the memory limit handling seems very odd. After running it's tests, it's also plain wrong. The test file is still STORE and not LZMA. So it's test don't actually test LZMA, but STORE. Against real LZMA encoded ZIP files, it breaks horribly. Stream::new_lzma_decoder() creates a decoder for .lzma file, which are not used in the ZIP spec (liblzma will try to parse a hader, that isn't there).

[Pr0methean]

Looks good, apart from one comment that needs rephrasing.

[hasenbanck]

@Pr0methean Great! You can ping me if LZMA, XZ or PPMd related issues should pop up in the future.