Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/cibuildwheel.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ jobs:
actions: read
with:
wheel-name-pattern: "srsly-*.whl"
pure-python: false
pure-python: true
create-release: ${{ startsWith(github.ref, 'refs/tags/release-') || startsWith(github.ref, 'refs/tags/prerelease-') }}
secrets:
gh-token: ${{ secrets.GITHUB_TOKEN }}
57 changes: 23 additions & 34 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,17 +14,26 @@ concurrency:
cancel-in-progress: true

env:
MODULE_NAME: 'srsly'
RUN_MYPY: 'false'

jobs:
tests:
name: ${{ matrix.python_version }} ${{ matrix.os }} numpy=${{ matrix.numpy }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
# FIXME: ujson segfault on 3.14
python_version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
# Note: ruamel.yaml does not support Python 3.13t
python_version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14", "3.14t"]
numpy: [true]
include:
- os: ubuntu-latest
python_version: "3.9"
numpy: false
- os: ubuntu-latest
python_version: "3.14"
numpy: false

runs-on: ${{ matrix.os }}

steps:
Expand All @@ -35,46 +44,26 @@ jobs:
uses: actions/setup-python@v6
with:
python-version: ${{ matrix.python_version }}
architecture: x64

- name: Build sdist
run: |
python -m pip install -U build pip setuptools
python -m pip install -U -r requirements.txt
python -m build --sdist

- name: Run mypy
shell: bash
if: ${{ env.RUN_MYPY == 'true' }}
run: |
python -m mypy $MODULE_NAME

- name: Delete source directory
shell: bash
run: |
rm -rf $MODULE_NAME
python -m pip install mypy
python -m mypy srsly

- name: Uninstall all packages
run: |
python -m pip freeze > installed.txt
python -m pip uninstall -y -r installed.txt
- name: Install package
run: python -m pip install .

- name: Install from sdist
- name: Test that numpy was not installed
shell: bash
run: |
SDIST=$(python -c "import os;print(os.listdir('./dist')[-1])" 2>&1)
python -m pip install dist/$SDIST
run: if pip list | grep -q '^numpy'; then exit 1; fi

- name: Test import
shell: bash
run: |
python -c "import $MODULE_NAME" -Werror
- name: Install numpy
if: ${{ matrix.numpy }}
run: python -m pip install numpy

- name: Install test requirements
run: |
python -m pip install -U -r requirements.txt
run: python -m pip install pytest

- name: Run tests
shell: bash
run: |
python -m pytest --pyargs $MODULE_NAME -Werror
run: pytest
Copy link
Contributor Author

@crusaderky crusaderky Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On 3.14t this emits

=============================== warnings summary ===============================
<frozen importlib._bootstrap>:491
  <frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'ujson', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================== 85 passed, 1 warning in 0.31s =========================

I'll do a one-liner follow-up that re-adds -Werror as soon as upstream releases of msgpack and ujson become available.

4 changes: 0 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
.env/
.env*
.vscode/
cythonize.json

# Byte-compiled / optimized / DLL files
__pycache__/
Expand Down Expand Up @@ -108,8 +107,5 @@ venv.bak/
# mypy
.mypy_cache/

# Cython intermediate files
*.cpp

# Vim files
*.sw*
42 changes: 17 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# srsly: Modern high-performance serialization utilities for Python

This package bundles some of the best Python serialization libraries into one
standalone package, with a high-level API that makes it easy to write code
convenience package, with a high-level API that makes it easy to write code
that's correct across platforms and Pythons. This allows us to provide all the
serialization utilities we need in a single binary wheel. Currently supports
**JSON**, **JSONL**, **MessagePack**, **Pickle** and **YAML**.
Expand All @@ -24,31 +24,24 @@ wrap the multiple serialization formats we need to support (especially `json`,
`msgpack` and `pickle`). These wrapping functions ended up duplicated across our
codebases, so we wanted to put them in one place.

At the same time, we noticed that having a lot of small dependencies was making
maintenance harder, and making installation slower. To solve this, we've made
`srsly` standalone, by including the component packages directly within it. This
way we can provide all the serialization utilities we need in a single binary
wheel.

`srsly` currently includes forks of the following packages:
`srsly` currently includes wrappers around the following packages:

- [`ujson`](https://github.com/esnme/ultrajson)
- [`msgpack`](https://github.com/msgpack/msgpack-python)
- [`msgpack-numpy`](https://github.com/lebedov/msgpack-numpy)
- [`cloudpickle`](https://github.com/cloudpipe/cloudpickle)
- [`ruamel.yaml`](https://github.com/pycontribs/ruamel-yaml) (without unsafe
implementations!)

## Installation
Additionally, it includes a heavily customized fork of
[`msgpack-numpy`](https://github.com/lebedov/msgpack-numpy), with corrected
round-trip behaviour for np.float64 objects.

> ⚠️ Note that `v2.x` is only compatible with **Python 3.6+**. For 2.7+
> compatibility, use `v1.x`.

`srsly` can be installed from pip. Before installing, make sure that your `pip`,
`setuptools` and `wheel` are up to date.
## Installation

`srsly` can be installed from pip.

```bash
python -m pip install -U pip setuptools wheel
python -m pip install srsly
```

Expand All @@ -58,12 +51,15 @@ Or from conda via conda-forge:
conda install -c conda-forge srsly
```

Alternatively, you can also compile the library from source. You'll need to make
sure that you have a development environment with a Python distribution
including header files, a compiler (XCode command-line tools on macOS / OS X or
Visual C++ build tools on Windows), pip and git installed.
This will automatically install/upgrade all dependencies.

Install from source:
numpy and cupy are optional dependencies for msgpack.
If numpy is installed, numpy objects can be serialized.
If cupy is installed, cupy objects will be automaticaly converted
to numpy and then serialized.


Alternatively, you can also install the library from the repository:

```bash
# clone the repo
Expand All @@ -74,10 +70,7 @@ cd srsly
python -m venv .env
source .env/bin/activate

# update pip
python -m pip install -U pip setuptools wheel

# compile and install from source
# install from source
python -m pip install .
```

Expand All @@ -86,7 +79,6 @@ mode without build isolation:

```bash
# install in editable mode
python -m pip install -r requirements.txt
python -m pip install --no-build-isolation --editable .

# run test suite
Expand Down
62 changes: 1 addition & 61 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,63 +1,3 @@
[build-system]
requires = [
"setuptools",
"cython>=0.29.1",
]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[tool.cibuildwheel]
build = "*"
skip = [
"cp38*", # Obsolete
"cp314-*", # FIXME ujson segfaults
"cp314t-*", # TODO free-threading support (note: 3.13t is skipped by default)
]
test-skip = ""

archs = ["native"]

build-frontend = "default"
config-settings = {}
dependency-versions = "pinned"
environment = {}
environment-pass = []
build-verbosity = 0

before-all = ""
before-build = ""
repair-wheel-command = ""

test-command = ""
before-test = ""
test-requires = []
test-extras = []

container-engine = "docker"

manylinux-x86_64-image = "manylinux2014"
manylinux-i686-image = "manylinux2014"
manylinux-aarch64-image = "manylinux2014"
manylinux-ppc64le-image = "manylinux2014"
manylinux-s390x-image = "manylinux2014"
manylinux-pypy_x86_64-image = "manylinux2014"
manylinux-pypy_i686-image = "manylinux2014"
manylinux-pypy_aarch64-image = "manylinux2014"

musllinux-x86_64-image = "musllinux_1_2"
musllinux-i686-image = "musllinux_1_2"
musllinux-aarch64-image = "musllinux_1_2"
musllinux-ppc64le-image = "musllinux_1_2"
musllinux-s390x-image = "musllinux_1_2"


[tool.cibuildwheel.linux]
repair-wheel-command = "auditwheel repair -w {dest_dir} {wheel}"

[tool.cibuildwheel.macos]
repair-wheel-command = "delocate-wheel --require-archs {delocate_archs} -w {dest_dir} -v {wheel}"

[tool.cibuildwheel.windows]

[tool.cibuildwheel.pyodide]


8 changes: 0 additions & 8 deletions requirements.txt

This file was deleted.

24 changes: 9 additions & 15 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -12,27 +12,26 @@ classifiers =
Intended Audience :: Developers
Intended Audience :: Science/Research
License :: OSI Approved :: MIT License
Operating System :: POSIX :: Linux
Operating System :: MacOS :: MacOS X
Operating System :: Microsoft :: Windows
Programming Language :: Cython
Operating System :: OS Independent
Programming Language :: Python :: 3
Programming Language :: Python :: 3.9
Programming Language :: Python :: 3.10
Programming Language :: Python :: 3.11
Programming Language :: Python :: 3.12
Programming Language :: Python :: 3.13
Programming Language :: Python :: 3.14
Topic :: Scientific/Engineering

[options]
zip_safe = true
include_package_data = true
# FIXME ujson segfaults on 3.14
python_requires = >=3.9,<3.14
setup_requires =
cython>=0.29.1
python_requires = >=3.9
Copy link
Contributor Author

@crusaderky crusaderky Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed version cap as there is very little that can break anymore with future versions. Future compatibility is now delegated to upstream libraries.

install_requires =
catalogue>=2.0.3,<2.1.0
catalogue>=2.0.10,<3
cloudpickle >=3.1.2,<4
msgpack >=1.1,<2
ruamel.yaml >=0.18.16,<1
ujson >=5.11.0,<6
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lower bounds are as of today. This is simply out of laziness to save the effort of pinpointing which is the minimum version of everything that works with thinc and spacy.

Higher bounds have been set very generously to reduce maintenance burden in srsly.


[options.entry_points]
# If spaCy is installed in the same environment as srsly, it will automatically
Expand All @@ -44,7 +43,7 @@ spacy_readers =
srsly.read_msgpack.v1 = srsly:read_msgpack

[bdist_wheel]
universal = false
universal = true

[sdist]
formats = gztar
Expand All @@ -55,11 +54,6 @@ max-line-length = 80
select = B,C,E,F,W,T4,B9
exclude =
srsly/__init__.py
srsly/msgpack/__init__.py
srsly/cloudpickle/__init__.py

[mypy]
ignore_missing_imports = True

[mypy-srsly.cloudpickle.*]
ignore_errors=True
Loading