Skip to content
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
96058cc
Start a discussion on helping downstream packaging
mgorny Jan 27, 2025
b4cb7d2
Add a section on Internet access
mgorny Jan 27, 2025
df7081c
Also suggest the option of splitting test data into separate archive
mgorny Jan 27, 2025
cff76df
Add a section on system dependencies
mgorny Jan 28, 2025
be242a8
Add a section on downstream testing
mgorny Jan 28, 2025
9f9af53
Expand on downstream testing, and add emphasis for readability
mgorny Jan 29, 2025
00f39e5
Reorganize into why/how sections, and add emphasis
mgorny Jan 29, 2025
08c70e8
Elaborate a bit more on why it's good to help downstreams
mgorny Jan 29, 2025
24f552e
Correct "pytest" capitalization
mgorny Jan 29, 2025
24b0346
Apply suggestions from code review
mgorny Jan 30, 2025
16288af
Apply more suggestions from code review
mgorny Jan 30, 2025
b4c7485
Attempt addressing the remaining review comments
mgorny Jan 30, 2025
df0c91a
Add a section on stable channels
mgorny Jan 30, 2025
fc72a38
Retitle as "Supporting downstream packaging"
mgorny Jan 31, 2025
24462f4
Add a "not all-or-nothing" sentence
mgorny Jan 31, 2025
29cc38a
Add a note that downstreams can send patches to fix these issues
mgorny Jan 31, 2025
9d5fbe6
Capitalize Git, per @pawamoy
mgorny Feb 1, 2025
4d95da2
Fix inconsistent case in bullet points and remove duplicate
mgorny Feb 1, 2025
6f55709
Apply typo fixes, thanks to @pawamoy
mgorny Feb 1, 2025
548ab34
Clarify that source distribution needs only package's files
mgorny Feb 1, 2025
e925da1
Fix inconsistent whitespace between sentences
mgorny Feb 1, 2025
0eb407c
Make the point of reusing source distribution lighter
mgorny Feb 1, 2025
94743f9
Clarify the Internet part
mgorny Feb 1, 2025
704d1a5
Apply suggestions from code review
mgorny Feb 1, 2025
e596609
Remove duplicate paragraph
mgorny Feb 1, 2025
169281d
Clarify source distributions
mgorny Feb 1, 2025
bb8ac35
Add non-reproducibility argument for changing resources
mgorny Feb 1, 2025
addf891
Mention removing duplication of patches and inconsistency
mgorny Feb 2, 2025
8a3a56c
Reword installing tests to make it clearer
mgorny Feb 2, 2025
58eaf85
Give an example of "catastrophic failure"
mgorny Feb 2, 2025
76aaf79
Indicate that some distributions require building from sources
mgorny Feb 22, 2025
4f97860
Merge branch 'main' into discussion-downstream
ncoghlan Feb 26, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Clarify the Internet part
Clarify the "do not use the Internet" part to make it clearer that we're
talking about backend actions, and custom build scripts in particular.
We're not talking of Python dependencies that are fetched and installed
by frontends.  Also, reflow the text to be more logical.
  • Loading branch information
mgorny committed Feb 1, 2025
commit 94743f973b2b68af2c3d16614e8aa4ab4edc52c4
40 changes: 26 additions & 14 deletions source/discussions/downstream-packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,12 @@ Why?
~~~~

Downstream builds are frequently done in sandboxed environments that cannot
access the Internet. Even if this is not the case, and assuming that you took
sufficient care to properly authenticate downloads, using the Internet
is discouraged for a number of reasons:
access the Internet. The package sources are unpacked into this environment,
and all the necessary dependencies are installed.

Even if this is not the case, and assuming that you took sufficient care to
properly authenticate downloads, using the Internet is discouraged for a number
of reasons:

- The Internet connection may be unstable (e.g. due to poor reception)
or suffer from temporary problems that could cause the process to fail
Expand All @@ -159,22 +162,31 @@ is discouraged for a number of reasons:
How?
~~~~

Your source distribution should either include all the files needed
for the package to build, or allow provisioning them externally. Ideally,
it should not even attempt to access the Internet at all, unless explicitly
requested to. If that is not possible to achieve, the next best thing
is to provide an opt-out switch to disable all Internet access.
If the package is implementing any custom build *backend* actions that use
the Internet, for example by automatically downloading vendored dependencies
or fetching Git submodules, its source distribution should either include all
of these files or allow provisioning them externally, and the Internet must not
be used if the files are already present.

Note that this point does not apply to Python dependencies that are specified
in the package metadata, and are fetched during the build and installation
process by *frontends* (such as :ref:`build` or :ref:`pip`). Downstreams use
frontends that use local provisioning for Python dependencies.

When such a switch is used, the build process should fail if some
of the required files are missing, rather than try to fetch them automatically.
This could be done e.g. by checking whether a ``NO_NETWORK`` environment
variable is set to a non-empty value. Please also remember that if you are
fetching remote resources, you must *verify their authenticity* (usually against
a hash), to protect against the file being substituted by a malicious party.
Ideally, custom build scripts should not even attempt to access the Internet
at all, unless explicitly requested to. If any resources are missing and need
to be fetched, they should ask the user for permission first. If that is not
feasible, the next best thing is to provide an opt-out switch to disable
all Internet access. This could be done e.g. by checking whether
a ``NO_NETWORK`` environment variable is set to a non-empty value.

Since downstreams frequently also run tests and build documentation, the above
should ideally extend to these processes as well.

Please also remember that if you are fetching remote resources, you absolutely
must *verify their authenticity* (usually against a hash), to protect against
the file being substituted by a malicious party.


.. _support-system-dependencies-in-builds:

Expand Down