Update to XZ 5.8.1 on Windows and macOS

[emmatyping]

The current version of XZ used by Windows and macOS (5.2.x) is no longer supported. We should update to 5.8.1 so that we are on a supported branch.

The macOS installer uses 5.2.3

cpython/Mac/BuildScript/build-installer.py

Lines 314 to 324 in a17c57e

	if PYTHON_3:
	result.extend([
	dict(
	name="XZ 5.2.3",
	url="http://tukaani.org/xz/xz-5.2.3.tar.gz",
	checksum='ef68674fb47a8b8e741b34e429d86e9d',
	configure_pre=[
	'--disable-dependency-tracking',
	]
	),
	])

The Windows build uses 5.2.5

cpython/PCbuild/get_externals.bat

Line 62 in a17c57e

set libraries=%libraries% xz-5.2.5

I have a draft PR to cpython-source-deps to import 5.8.1 python/cpython-source-deps#44, but given the large version change the MSVC build definition will likely need some updating too (see e.g. #140818).

Linked PRs

• gh-140849: Update macOS installer to XZ 5.8.1 #140858
• gh-140849: Update bundled liblzma to 5.8.1 on Windows #141022

[zooba]

I have a draft PR to cpython-source-deps to import 5.8.1

We don't use PRs on that repo. A "trusted person" (a.k.a. core developer) has to validate the source package by hand anyway, so they may as well download it, check the hash, import/tag/push themselves. A PR just forces them to do extra steps.

Leave a request somewhere we can see it (like here) and it'll get done. A link to the official download page is helpful.

[emmatyping]

@zooba right! Forgot about that, thanks for the reminder. The official release page for 5.8.1 is https://github.com/tukaani-project/xz/releases/tag/v5.8.1

[zooba]

Great. Which parts do we need? The licensing is a bit complicated, and would be good to filter all the GPL code out before we check it in.

[zooba]

Looks like we need to generate a config.h and the CRC tables? I can do those, I think.

[zooba]

With a config.h (that I've had to create myself from their CMake setup) and a few updates to our liblzma project file, I get a build but a range of test errors like these:

======================================================================
ERROR: test_roundtrip (test.test_lzma.CompressDecompressFunctionTestCase.test_roundtrip)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\cpython\Lib\test\test_lzma.py", line 501, in test_roundtrip
    cdata = lzma.compress(INPUT, lzma.FORMAT_RAW, filters=FILTERS_RAW_4)
  File "D:\cpython\Lib\lzma.py", line 335, in compress
    comp = LZMACompressor(format, check, preset, filters)
_lzma.LZMAError: Invalid or unsupported options

Since everything builds, I don't think it's going to be config options, but I don't want to commit and tag the sources (which will include our config.h, because that's how it's designed) until this is clean.

Anyone have any ideas? I'll get back to looking tomorrow, but it's late here now so I'm quitting for the night (and then I'm away the rest of the week, so if there are no ideas tomorrow, anyone who wants to look at doing their own build can take their time).

The config.h I'm using right now is config.h.txt, but that probably won't be suitable for our cross-platform builds, so don't commit it as-is. I'm getting those errors on an x64 build, and the config is based on their x64 settings/my x64 laptop, so it should be fine (identical results for Win32 build). I'll deal with the cross-platform builds once the first one works.

[emmatyping]

@zooba looks like you need to enable delta encoding (and probably a few other encoders). Based on the current config.h there are a number of encoders/decoders that are enabled there that aren't in your config.h. These are not platform enable/disable functions but rather routines to compress bcj or branch-call-jump in compression input data (e.g. LZMA can rewrite bcjs to reduce the size of the executable). Therefore these should be enabled even though Windows will only run x86 or ARM code. There's also delta compression which is meant for small updates to files as the name implies, that leads to the test example you mention failing (the lzma.FILTER_DELTA is the unsupported option)

[zooba]

I've got it working. Just verifying on other Windows architectures and then will push changes.