Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
96058cc
Start a discussion on helping downstream packaging
mgorny Jan 27, 2025
b4cb7d2
Add a section on Internet access
mgorny Jan 27, 2025
df7081c
Also suggest the option of splitting test data into separate archive
mgorny Jan 27, 2025
cff76df
Add a section on system dependencies
mgorny Jan 28, 2025
be242a8
Add a section on downstream testing
mgorny Jan 28, 2025
9f9af53
Expand on downstream testing, and add emphasis for readability
mgorny Jan 29, 2025
00f39e5
Reorganize into why/how sections, and add emphasis
mgorny Jan 29, 2025
08c70e8
Elaborate a bit more on why it's good to help downstreams
mgorny Jan 29, 2025
24f552e
Correct "pytest" capitalization
mgorny Jan 29, 2025
24b0346
Apply suggestions from code review
mgorny Jan 30, 2025
16288af
Apply more suggestions from code review
mgorny Jan 30, 2025
b4c7485
Attempt addressing the remaining review comments
mgorny Jan 30, 2025
df0c91a
Add a section on stable channels
mgorny Jan 30, 2025
fc72a38
Retitle as "Supporting downstream packaging"
mgorny Jan 31, 2025
24462f4
Add a "not all-or-nothing" sentence
mgorny Jan 31, 2025
29cc38a
Add a note that downstreams can send patches to fix these issues
mgorny Jan 31, 2025
9d5fbe6
Capitalize Git, per @pawamoy
mgorny Feb 1, 2025
4d95da2
Fix inconsistent case in bullet points and remove duplicate
mgorny Feb 1, 2025
6f55709
Apply typo fixes, thanks to @pawamoy
mgorny Feb 1, 2025
548ab34
Clarify that source distribution needs only package's files
mgorny Feb 1, 2025
e925da1
Fix inconsistent whitespace between sentences
mgorny Feb 1, 2025
0eb407c
Make the point of reusing source distribution lighter
mgorny Feb 1, 2025
94743f9
Clarify the Internet part
mgorny Feb 1, 2025
704d1a5
Apply suggestions from code review
mgorny Feb 1, 2025
e596609
Remove duplicate paragraph
mgorny Feb 1, 2025
169281d
Clarify source distributions
mgorny Feb 1, 2025
bb8ac35
Add non-reproducibility argument for changing resources
mgorny Feb 1, 2025
addf891
Mention removing duplication of patches and inconsistency
mgorny Feb 2, 2025
8a3a56c
Reword installing tests to make it clearer
mgorny Feb 2, 2025
58eaf85
Give an example of "catastrophic failure"
mgorny Feb 2, 2025
76aaf79
Indicate that some distributions require building from sources
mgorny Feb 22, 2025
4f97860
Merge branch 'main' into discussion-downstream
ncoghlan Feb 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Attempt addressing the remaining review comments
  • Loading branch information
mgorny committed Jan 30, 2025
commit b4c7485665311525bf50951f6ecf50ff541caa18
32 changes: 15 additions & 17 deletions source/discussions/downstream-packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,10 @@ a few important reasons to provide a static archive file instead:
supported than e.g. using a git clone. This can help users with poor
Internet connectivity.

- Downstreams often **use checksums to verify the authenticity** of source files
- Downstreams often use hashes to verify the authenticity of source files
on subsequent builds, which require that they remain bitwise identical over
time. For example, automatically generated git archives do not guarantee
that.
this, as the compressed data may change if gzip is upgraded on the server.

- Archive files can be mirrored, reducing both upstream and downstream
bandwidth use. The actual builds can afterwards be performed in firewalled
Expand Down Expand Up @@ -132,8 +132,8 @@ is discouraged for a number of reasons:
unavailable, making the build no longer possible. This is especially
problematic when someone needs to build an old package version.

- Accessing remote servers poses a **privacy** issue and a potential
**security** issue, as it exposes information about the system building
- Accessing remote servers poses a privacy issue and a potential
security issue, as it exposes information about the system building
the package.

- The user may be using a service with a limited data plan, in which
Expand All @@ -153,8 +153,8 @@ When such a switch is used, the build process should fail if some
of the required files are missing, rather than try to fetch them automatically.
This could be done e.g. by checking whether a ``NO_NETWORK`` environment
variable is set to a non-empty value. Please also remember that if you are
fetching remote resources, you must **verify their authenticity**, e.g. against
a checksum, to protect against the file being substituted by a malicious party.
fetching remote resources, you must *verify their authenticity* (usually against
a hash), to protect against the file being substituted by a malicious party.

Since downstreams frequently also run tests and build documentation, the above
should ideally extend to these processes as well.
Expand All @@ -165,11 +165,9 @@ should ideally extend to these processes as well.
Support building against system dependencies
--------------------------------------------


Why?
~~~~


Some Python projects have non-Python dependencies, such as libraries written
in C or C++. Trying to use the system versions of these dependencies
in upstream packaging may cause a number of problems for end users:
Expand Down Expand Up @@ -237,7 +235,6 @@ In particular:
How?
~~~~


A good compromise between the needs of both parties is to provide a switch
between using vendored and system dependencies. Ideally, if the package has
multiple vendored dependencies, it should provide both individual switches
Expand Down Expand Up @@ -266,11 +263,9 @@ are better equipped to handle.
Support downstream testing
--------------------------


Why?
~~~~


A variety of downstream projects run some degree of testing on the packaged
Python projects. Depending on the particular case, this can range from minimal
smoke testing to comprehensive runs of the complete test suite. There can
Expand All @@ -287,7 +282,7 @@ be various reasons for doing this, for example:
the ones present during upstream release testing.

- Testing the package in an environment closely resembling the production
setup. This can detect issues caused by nontrivial interactions between
setup. This can detect issues caused by non-trivial interactions between
different installed packages, including packages that are not dependencies
of your package, but nevertheless can cause issues.

Expand All @@ -297,15 +292,14 @@ be various reasons for doing this, for example:
Admittedly, sometimes downstream testing may yield false positives or bug
reports about scenarios the upstream project is not interested in supporting.
However, perhaps even more often it does provide early notice of problems,
or find nontrivial bugs that would otherwise cause issues for your users
in production. And believe me, the majority of **downstream packagers are doing
their best to double-check their results, and help you triage and fix the bugs
that they report**.
or find non-trivial bugs that would otherwise cause issues for the upstream
project's users. While mistakes do happen, the majority of downstream packagers
are doing their best to double-check their results, and help upstream
maintainers triage and fix the bugs that they reported.

How?
~~~~


There are a number of things that upstream projects can do to help downstream
repackagers test their packages efficiently and effectively, including some of the suggestions
already mentioned above. These are typically improvements that make the test suite more
Expand Down Expand Up @@ -367,6 +361,10 @@ Some specific suggestions are:
the ability to conveniently deselect tests, rerun flaky tests
(via pytest-rerunfailures_), add a timeout to prevent tests from hanging
(via pytest-timeout_) or run tests in parallel (via pytest-xdist_).
Note that test suites don't need to be *written* with ``pytest`` to be
*executed* with ``pytest``: ``pytest`` is able to find and execute almost
all test cases that are compatible with the standard library's ``unittest``
test discovery.


.. _responses: https://pypi.org/project/responses/
Expand Down