Skip to content

feat: Prefilter Shard Records on Python Minor Version#4214

Open
jjerphan wants to merge 13 commits intomamba-org:mainfrom
jjerphan:feat/filter-shard-record-python-minor-version
Open

feat: Prefilter Shard Records on Python Minor Version#4214
jjerphan wants to merge 13 commits intomamba-org:mainfrom
jjerphan:feat/filter-shard-record-python-minor-version

Conversation

@jjerphan
Copy link
Copy Markdown
Member

@jjerphan jjerphan commented Mar 26, 2026

Description

Comes after #4187 and #4220.

Since the minor versions python is pinned in users' environments, we can prefilter records in shards using it to only selected relevant builds.

This definitely eases the installation of packages depending on python, by reducing the number of clauses within the solver and thus the time spent and RAM used resolving the environments. For instance for pyarrow, it reduces the time spent by 5 and the memory footprint by 20.

If python is not installed, we first try to prefilter builds depending on a fallback minor version (python 3.14 proposed), before potentially retrying without (current behavior) if this fallback minor version cannot be used.

Type of Change

  • Bugfix
  • Feature / enhancement
  • CI / Documentation
  • Maintenance

Checklist

  • My code follows the general style and conventions of the codebase, ensuring consistency
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • I have run pre-commit run --all locally in the source folder and confirmed that there are no linter errors.
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing tests pass locally with my changes

@github-actions github-actions bot added the release::enhancements For enhancements PRs or implementing features label Mar 26, 2026
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch from 2a15510 to 02817f8 Compare March 26, 2026 15:30
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 26, 2026

Codecov Report

❌ Patch coverage is 44.44444% with 95 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.72%. Comparing base (39702c7) to head (a4e0425).

Files with missing lines Patch % Lines
libmamba/src/api/utils.cpp 15.71% 59 Missing ⚠️
libmamba/src/api/install.cpp 0.00% 15 Missing ⚠️
libmamba/src/core/shards.cpp 75.00% 9 Missing ⚠️
libmamba/src/api/channel_loader.cpp 33.33% 6 Missing ⚠️
libmamba/src/api/update.cpp 0.00% 3 Missing ⚠️
libmamba/src/core/shard_python_minor_prefilter.cpp 92.10% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4214      +/-   ##
==========================================
- Coverage   53.65%   52.72%   -0.93%     
==========================================
  Files         239      240       +1     
  Lines       29349    29510     +161     
  Branches     3122     3151      +29     
==========================================
- Hits        15747    15559     -188     
- Misses      13599    13948     +349     
  Partials        3        3              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch 2 times, most recently from 9c47b94 to 51f2501 Compare March 27, 2026 09:25
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch 5 times, most recently from afb6f6a to 55f2328 Compare March 31, 2026 09:55
@jjerphan jjerphan marked this pull request as ready for review March 31, 2026 09:56
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch 3 times, most recently from cc572b0 to 9e2aa2b Compare April 2, 2026 09:24
@jjerphan jjerphan marked this pull request as draft April 2, 2026 13:50
@jjerphan jjerphan marked this pull request as draft April 2, 2026 13:50
@jjerphan jjerphan marked this pull request as draft April 2, 2026 13:50
@jjerphan jjerphan marked this pull request as draft April 2, 2026 13:50
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch from 9b1f296 to 5a2da95 Compare April 2, 2026 14:04
jjerphan added 7 commits April 2, 2026 17:39
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Such as:

```
warning  libmamba Failed to parse field 'noarch' (msgpack type=1) in shard package record for 'tensorboard-2.1.1-py38_0.tar.bz2': Expected STR or BIN type for string conversion. This field will be ignored.
```

Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch from 0a72888 to ea75e1d Compare April 2, 2026 15:42
@jjerphan jjerphan marked this pull request as ready for review April 2, 2026 16:11
@jjerphan jjerphan changed the title feat: Filter Shard Record on Python Minor Version feat: Prefilter Shard Records on Python Minor Version Apr 2, 2026
}

std::optional<specs::Version>
installed_python_minor_for_prefix(const fs::u8path& target_prefix)
Copy link
Copy Markdown
Member Author

@jjerphan jjerphan Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there utilities we could reuse instead?

Copy link
Copy Markdown
Member Author

@jjerphan jjerphan Apr 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PrefixData could be reused, but this would be much more costly (because it parses all the installed packages JSON and convert them to PackageInfo). We only need one specs::Version here.

jjerphan added 2 commits April 3, 2026 09:30
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch from 3f4ee57 to fb64ec3 Compare April 3, 2026 07:32
jjerphan added 4 commits April 3, 2026 09:55
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
Signed-off-by: Julien Jerphanion <git@jjerphan.xyz>
@jjerphan jjerphan force-pushed the feat/filter-shard-record-python-minor-version branch from d388860 to a4e0425 Compare April 3, 2026 07:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release::enhancements For enhancements PRs or implementing features

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant