Skip to content

Keep the Linux x86_64 cp312 wheel under PyPI's 100 MB limit #382

@Trinkle23897

Description

@Trinkle23897

Summary

The Linux x86_64 cp312 wheel is now too large to upload to PyPI.

Concrete blocker:

  • Wheel: envpool-1.1.0-cp312-cp312-manylinux_2_28_x86_64.whl
  • Current size: 117,913,151 bytes
  • PyPI project file-size limit for envpool: 100 MB
  • Upload failure: 400 File too large. Limit for project 'envpool' is 100 MB

This means the current release workflow can successfully build the wheel, but the PyPI publish step cannot ship it.

Current observations

A few measurements from the 1.1.0 Linux cp312 x86_64 wheel:

  • Running strip --strip-unneeded on the EnvPool-built Linux .so files only reduces the wheel from 117,913,151 to 115,440,654 bytes, so strip-only is not enough.
  • The biggest contributors include:
    • envpool/vizdoom/bin/freedoom2.wad (28.5 MB raw)
    • envpool.libs/libicudata-*.so (26.9 MB raw)
    • large vendored Qt/ICU libraries pulled into envpool.libs
    • mujoco/robotics/assets/**/* (~25.9 MB compressed total)
  • There are also confirmed dead robotics assets in the current wheel that are not referenced by the currently registered robotics envs, worth about ~12.7 MB compressed in total.

Possible optimization directions

This issue is intentionally not prescribing a single fix yet. Reasonable directions include:

  1. Strip Linux-built shared objects in the release build.
  • This should be done regardless, but it is only a partial improvement.
  1. Remove confirmed unused robotics assets from the wheel.
  • There are currently dead files under the robotics asset tree that appear to be unused by the registered env set.
  • This is a relatively low-risk size win.
  1. Move large third-party assets out of the core wheel.
  • Candidates include ViZDoom runtime assets (freedoom2.wad, vizdoom, maps) and large MuJoCo robotics assets.
  • Possible implementations:
    • runtime download with hash verification and local cache
    • explicit prefetch command
    • split asset package(s)
  1. Reduce or isolate the Linux-only vendored dependency payload.
  • In particular, procgen currently pulls Qt/ICU into envpool.libs on Linux.
  • If possible, either make that dependency smaller, make it optional, or split it out of the base wheel.

CI follow-up requested in this issue

After we have a fix or mitigation, we should add a dedicated wheel-size guard in CI:

  • Run the Linux x86_64 py3.12 release-wheel build on every commit / PR update, not only on main.
  • Do not run the full release matrix for this guard; only run the single release-linux (x86_64, py3.12) build.
  • After building wheelhouse/*.whl, assert that the produced wheel is strictly below PyPI's limit (< 100 MB).
  • Make that check fail loudly so regressions are caught before release time.

Why this matters

Right now, the failure is only discovered at release/publish time. A dedicated per-commit size guard for the exact problematic target would make this a normal CI regression instead of a release-day surprise.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions