From 2a274d826c7e031e3d05659a0888c395994dadd9 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 9 Nov 2020 14:01:21 -0800 Subject: [PATCH 001/793] Add Batuhan Taskaya to the developer log --- developers.csv | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/developers.csv b/developers.csv index 561775364..48cc31309 100644 --- a/developers.csv +++ b/developers.csv @@ -1,3 +1,4 @@ +Batuhan Taskaya,isidentical,2020-11-08,, Brandt Bucher,brandtbucher,2020-09-14,, Lysandros Nikolaou,lysnikolaou,2020-06-29,, Kyle Stanley,aeros,2020-04-14,, @@ -125,7 +126,7 @@ Brett Cannon,brettcannon,2003-04-18,, David Goodger,,2003-01-02,2017-02-10,Did not make GitHub transition Gustavo Niemeyer,,2002-11-05,2017-02-10,Did not make GitHub transition Tony Lownds,,2002-09-22,2017-02-10,Did not make GitHub transition -Steve Holden,,2002-06-14,2017-02-10,"Relinquished privileges on 2005-04-07, +Steve Holden,,2002-06-14,2017-02-10,"Relinquished privileges on 2005-04-07, but granted again for Need for Speed sprint; did not make GitHub transition" Christian Tismer,ctismer,2002-05-17,,For Need for Speed sprint Jason Tishler,,2002-05-15,2017-02-10,Did not make GitHub transition From eb2359d3870b8e05c1c17924dc2c0b440fa1a7e1 Mon Sep 17 00:00:00 2001 From: Batuhan Taskaya Date: Tue, 10 Nov 2020 02:10:45 +0300 Subject: [PATCH 002/793] Add myself to the experts index (#642) --- experts.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/experts.rst b/experts.rst index 52da0c987..473e9f997 100644 --- a/experts.rst +++ b/experts.rst @@ -55,7 +55,7 @@ abc aifc r.david.murray argparse rhettinger* array -ast benjamin.peterson, pablogsal +ast benjamin.peterson, pablogsal, BTaskaya asynchat josiahcarlson, giampaolo.rodola*, stutzbach asyncio yselivanov, asvetlov asyncore josiahcarlson, giampaolo.rodola*, stutzbach @@ -182,7 +182,7 @@ pstats pty twouters* pwd py_compile -pyclbr +pyclbr BTaskaya pydoc queue rhettinger* quopri @@ -310,7 +310,7 @@ Interest Area Maintainers ================== ========================================================== algorithms rhettinger* argument clinic larry -ast/compiler benjamin.peterson, brett.cannon, yselivanov, pablogsal, Mark.Shannon +ast/compiler benjamin.peterson, brett.cannon, yselivanov, pablogsal, Mark.Shannon, BTaskaya autoconf/makefiles twouters* bsd bug tracker ezio.melotti From b6d8676d911bfb915f43003ce957cad142dba476 Mon Sep 17 00:00:00 2001 From: Abdur-Rahmaan Janhangeer Date: Wed, 11 Nov 2020 03:55:28 +0400 Subject: [PATCH 003/793] Changing Arabic coordinator (#641) --- documenting.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documenting.rst b/documenting.rst index ab96e7f51..b1a3151d8 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1595,7 +1595,7 @@ in production, other are work in progress: +-----------------+-------------------------------+----------------------------+ | Language | Contact | Links | +=================+===============================+============================+ -| Arabic (ar) | Abdur-Rahmaan Janhangeer | `GitHub `_ | +| Arabic (ar) | Ibrahim Elbouhissi | `GitHub `_ | +-----------------+-------------------------------+----------------------------+ | Bengali as | `Kushal Das `_ | `GitHub `_ | | spoken in | | | From bc37079135d8c3b30f7ddf3d42bcf3beea82c3cc Mon Sep 17 00:00:00 2001 From: E-Paine <63801254+E-Paine@users.noreply.github.com> Date: Tue, 1 Dec 2020 19:33:13 +0000 Subject: [PATCH 004/793] Improve instruction for updating with upstream (#646) Co-authored-by: Mariatta --- gitbootcamp.rst | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/gitbootcamp.rst b/gitbootcamp.rst index 7a8405a22..bd5f8e441 100644 --- a/gitbootcamp.rst +++ b/gitbootcamp.rst @@ -237,8 +237,8 @@ Creating a Pull Request 6. Press the ``Create pull request`` button. -Syncing with Upstream ---------------------- +Updating your CPython Fork +-------------------------- Scenario: @@ -249,12 +249,19 @@ Scenario: - You now want to update your forked CPython repository to be the same as the upstream CPython repository. +Please do not try to solve this by creating a pull request from +``python:master`` to ``:master`` as the authors of the patches will +get notified unnecessarily. + Solution:: git checkout master git pull upstream master git push origin master +.. note:: For the above commands to work, please follow the instructions found + in the :ref:`checkout` section + Another scenario: - You created ``some-branch`` some time ago. From e5918c0a0558d1c530a49c916fb83de211faca2e Mon Sep 17 00:00:00 2001 From: larryhastings Date: Thu, 17 Dec 2020 09:29:22 -0800 Subject: [PATCH 005/793] Update Larry's status in the Administrators list (#636) --- devcycle.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/devcycle.rst b/devcycle.rst index 858fa5514..06c7eaddd 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -319,7 +319,7 @@ Current Administrators +-------------------+----------------------------------------------------------+-----------------+ | Ned Deily | Python 3.6 and 3.7 Release Manager | ned-deily | +-------------------+----------------------------------------------------------+-----------------+ -| Lary Hastings | Python 3.5 Release Manager | larryhastings | +| Lary Hastings | Retired Release Manager (for Python 3.4 and 3.5) | larryhastings | +-------------------+----------------------------------------------------------+-----------------+ | Berker Peksag | Maintainer of bpo-linkify and cpython-emailer-webhook | berkerpeksag | +-------------------+----------------------------------------------------------+-----------------+ From 5fea5796c1926bda7f728b4099ccb9630cbf5913 Mon Sep 17 00:00:00 2001 From: Carol Willing Date: Sat, 19 Dec 2020 10:41:51 -0800 Subject: [PATCH 006/793] Update my motivations entry --- motivations.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/motivations.rst b/motivations.rst index e5ed47930..50085000c 100644 --- a/motivations.rst +++ b/motivations.rst @@ -257,9 +257,10 @@ participating in the CPython core development process: .. topic:: Carol Willing (United States) + * Noteable: ``__ (Technical Evangelist) * Personal site: `Willing Consulting `_ * `Extended bio `__ - * Project Jupyter, Cal Poly SLO (Research Software Engineer) + * Project Jupyter (Steering Council, Core Team for JupyterHub/Binder) * Python Software Foundation (Fellow) Carol is focused on Python's usage in education and scientific research. From bc8823bbfe82d8e8ef6abb15cd5876530b5679f9 Mon Sep 17 00:00:00 2001 From: jablonskidev <36643503+jablonskidev@users.noreply.github.com> Date: Mon, 21 Dec 2020 16:32:04 -0800 Subject: [PATCH 007/793] Made wording more welcoming (#649) Made wording more in the translation Q&A be more welcoming. Closes https://github.com/python/devguide/issues/522 Co-authored-by: Mariatta --- documenting.rst | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/documenting.rst b/documenting.rst index b1a3151d8..ebf5a58e2 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1792,10 +1792,9 @@ Ask on doc-sig, or better, make a PR on the `devguide I have a translation, but not on git, what should I do? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Just ask for help on the doc-sig mailing list and our python-fu, git-fu -and bash-fu combined will help you create an appropriate repository. If -you use a tool like transifex don’t worry. Keeping them in sync is not -that hard. +You can ask for help on the doc-sig mailing list, and the team will help you +create an appropriate repository. You can still use tools like transifex, +if you like. My git hierarchy does not match yours, can I keep it? From b39db20b309e16050625d035a968c9db7f6ddbaa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=D0=91=D0=BE=D1=80=D0=B8=D1=81=20=D0=92=D0=B5=D1=80=D1=85?= =?UTF-8?q?=D0=BE=D0=B2=D1=81=D0=BA=D0=B8=D0=B9?= Date: Mon, 25 Jan 2021 10:59:56 -0500 Subject: [PATCH 008/793] Minor corrections in garbage_collector.rst (#656) --- garbage_collector.rst | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/garbage_collector.rst b/garbage_collector.rst index 0f93be4af..68a4df028 100644 --- a/garbage_collector.rst +++ b/garbage_collector.rst @@ -219,17 +219,17 @@ they started originally) and setting its ``gc_refs`` field to 1. This is what ha to ``link_2`` and ``link_3`` below as they are reachable from ``link_1``. From the state in the previous image and after examining the objects referred to by ``link_1`` the GC knows that ``link_3`` is reachable after all, so it is moved back to the -original list and its ``gc_refs`` field is set to one so if the GC visits it again, it -does know that is reachable. To avoid visiting a object twice, the GC marks all -objects that are already visited once (by unsetting the ``PREV_MASK_COLLECTING`` flag) -so if an object that has already been processed is referred by some other object, the -GC does not process it twice. +original list and its ``gc_refs`` field is set to 1 so that if the GC visits it again, +it will know that it's reachable. To avoid visiting an object twice, the GC marks all +objects that have already been visited once (by unsetting the ``PREV_MASK_COLLECTING`` +flag) so that if an object that has already been processed is referenced by some other +object, the GC does not process it twice. .. figure:: images/python-cyclic-gc-5-new-page.png -Notice that once an object that was marked as "tentatively unreachable" and later is -moved back to the reachable list, it will be visited again by the garbage collector -as now all the references that that objects has need to be processed as well. This +Notice that an object that was marked as "tentatively unreachable" and was later +moved back to the reachable list will be visited again by the garbage collector +as now all the references that that object has need to be processed as well. This process is really a breadth first search over the object graph. Once all the objects are scanned, the GC knows that all container objects in the tentatively unreachable list are really unreachable and can thus be garbage collected. @@ -276,7 +276,7 @@ follows these steps in order: set is going to be destroyed and has weak references with callbacks, these callbacks need to be honored. This process is **very** delicate as any error can cause objects that will be in an inconsistent state to be resurrected or reached - by some python functions invoked from the callbacks. In addition, weak references + by some Python functions invoked from the callbacks. In addition, weak references that also are part of the unreachable set (the object and its weak reference are in cycles that are unreachable) need to be cleaned immediately, without executing the callback. Otherwise it will be triggered later, @@ -304,9 +304,9 @@ optimization: generations. The main idea behind this concept is the assumption t most objects have a very short lifespan and can thus be collected shortly after their creation. This has proven to be very close to the reality of many Python programs as many temporarily objects are created and destroyed very fast. The older an object is -the less likely it is to become unreachable. +the less likely it is that it will become unreachable. -To take advantage of this fact, all container objects are segregated across +To take advantage of this fact, all container objects are segregated into three spaces/generations. Every new object starts in the first generation (generation 0). The previous algorithm is executed only over the objects of a particular generation and if an object @@ -316,7 +316,7 @@ the same object survives another GC round in this new generation (generation 1) it will be moved to the last generation (generation 2) where it will be surveyed the least often. -Generations are collected when the number of objects that they contain reach some +Generations are collected when the number of objects that they contain reaches some predefined threshold, which is unique for each generation and is lower the older the generations are. These thresholds can be examined using the ``gc.get_threshold`` function: @@ -402,10 +402,10 @@ bit a separate tag) – as long as code that uses the pointer masks out these bits before accessing memory. E.g., on a 32-bit architecture (for both addresses and word size), a word is 32 bits = 4 bytes, so word-aligned addresses are always a multiple of 4, hence end in ``00``, leaving the last 2 bits -available; while on a 64-bit architecture, a word is 64 bits word = 8 bytes, so +available; while on a 64-bit architecture, a word is 64 bits = 8 bytes, so word-aligned addresses end in ``000``, leaving the last 3 bits available. -The CPython GC makes use of two fat pointers that corresponds to the extra fields +The CPython GC makes use of two fat pointers that correspond to the extra fields of ``PyGC_Head`` discussed in the `Memory layout and object structure`_ section: .. warning:: @@ -440,7 +440,7 @@ Optimization: delay tracking containers Certain types of containers cannot participate in a reference cycle, and so do not need to be tracked by the garbage collector. Untracking these objects -reduces the cost of garbage collections. However, determining which objects may +reduces the cost of garbage collection. However, determining which objects may be untracked is not free, and the costs must be weighed against the benefits for garbage collection. There are two possible strategies for when to untrack a container: @@ -470,7 +470,7 @@ benefit from delayed tracking: full garbage collection (all generations), the collector will untrack any dictionaries whose contents are not tracked. -The garbage collector module provides the python function is_tracked(obj), which returns +The garbage collector module provides the Python function ``is_tracked(obj)``, which returns the current tracking status of the object. Subsequent garbage collections may change the tracking status of the object. From 793046c8b8b8d3fc0fbcabe43aa1a01061494cf2 Mon Sep 17 00:00:00 2001 From: jablonskidev <36643503+jablonskidev@users.noreply.github.com> Date: Fri, 29 Jan 2021 16:23:35 -0800 Subject: [PATCH 009/793] Make https://devguide.python.org/committing/ simpler (#650) Closes #520 --- committing.rst | 390 ++++++++++++++++++++----------------------------- 1 file changed, 162 insertions(+), 228 deletions(-) diff --git a/committing.rst b/committing.rst index 773dd66cc..a242e8aa9 100644 --- a/committing.rst +++ b/committing.rst @@ -5,212 +5,147 @@ Accepting Pull Requests .. highlight:: none -This page is aimed to core developers, and covers the steps required to -accept, merge, and possibly backport a pull request on the main repository. - -Is the PR ready to be accepted? -------------------------------- - -Before a PR is accepted, you must make sure it is ready to enter the public -source tree. Use the following as a checklist of what to check for before -merging (details of various steps can be found later in this document): - -#. Has the submitter signed the CLA? - (delineated by a label on the pull request) -#. Did the test suite pass? (delineated by a pull request check) -#. Did code coverage increase or stay the same? - (delineated by a comment on the pull request) -#. Are the changes acceptable? -#. Was ``configure`` regenerated (if necessary)? -#. Was ``pyconfig.h.in`` regenerated (if necessary)? -#. Was the submitter added to ``Misc/ACKS`` (as appropriate)? -#. Was an entry added under ``Misc/NEWS.d/next`` (as appropriate)? -#. Was "What's New" updated (as appropriate)? -#. Were appropriate labels added to signify necessary backporting of the - pull request? -#. Was the pull request first made against the ``master`` branch? - -.. note:: - If you want to share your work-in-progress code on a feature or bugfix, - either open a ``WIP``-prefixed PR, publish patches on the - `issue tracker`_ or create a public fork of the repository. - -.. _issue tracker: https://bugs.python.org - - -Does the test suite still pass? -''''''''''''''''''''''''''''''' - -You must :ref:`run the whole test suite ` to ensure that it -passes before merging any code changes. - -.. note:: - You really need to run the **entire** test suite. Running a single test - is not enough as the changes may have unforeseen effects on other tests - or library modules. - - Running the entire test suite doesn't guarantee that the changes - will pass the :ref:`continuous integration ` tests, as those - will exercise more possibilities still (such as different platforms or - build options). But it will at least catch non-build specific, - non-platform specific errors, therefore minimizing the chance for - breakage. - - -Patch checklist -''''''''''''''' - -You should also :ref:`run patchcheck ` to perform a quick -sanity check on the changes. - - -Handling Others' Code ---------------------- - -As a core developer you will occasionally want to commit a patch created by -someone else. When doing so you will want to make sure of some things. - -First, make sure the patch is in a good state. Both :ref:`patch` and -:ref:`helptriage` -explain what is to be expected of a patch. Typically patches that get cleared by -triagers are good to go except maybe lacking ``Misc/ACKS`` and ``Misc/NEWS.d`` -entries (which a core developer should make sure are updated appropriately). - -Second, make sure the patch does not break backwards-compatibility without a -good reason. This means :ref:`running the entire test suite ` to -make sure everything still passes. It also means that if semantics do change -there must be a good reason for the breakage of code the change will cause -(and it **will** break someone's code). If you are unsure if the breakage -is worth it, ask on python-dev. - -Third, ensure the patch is attributed correctly with the contributor's -name in ``Misc/ACKS`` if they aren't already there (and didn't add themselves -in their patch) and by mentioning "Patch by " in the ``Misc/NEWS.d`` entry -and the check-in message. If the patch has been heavily modified then "Initial -patch by " is an appropriate alternate wording. GitHub now supports `multiple -authors `_ -in a commit. Add ``Co-authored-by: name `` at the end of the commit message. - -If you omit correct attribution in the initial check-in, then update ``ACKS`` -and ``NEWS.d`` in a subsequent check-in (don't worry about trying to fix the -original check-in message in that case). - -Finally, make sure that the submitter of the -patch has a CLA in place (indicated by an asterisk following their username -in the `issue tracker`_ or by the "CLA Signed" label on the pull request). -If the submitter lacks a signed CLA and the patch is non-trivial, direct them -to the electronic `Contributor Licensing Agreement`_ -to ensure the PSF has the appropriate authorizations in place to relicense -and redistribute their code. - - -Contributor Licensing Agreements --------------------------------- +This page is a step-by-step guide for core developers who need to assess, +merge, and possibly backport a pull request on the main repository. + +Assessing a pull request +------------------------ + +Before you can accept a pull request, you need to make sure that it is ready +to enter the public source tree. Ask yourself the following questions: + +* **Are there ongoing discussions at** ``bugs.python.org`` **(b.p.o.)?** + Read the linked b.p.o. issue. If there are ongoing discussions, then + we need to have a resolution there before we can merge the pull request. + +* **Was the pull request first made against the appropriate branch?** + The only branch that receives new features is ``master``, the + in-development branch. Pull requests should only target bug-fix branches + if an issue appears in only that version and possibly older versions. + +* **Are there comments on the pull request?** + Look for explanations about whether the code coverage increased or + stayed the same. + +* **Are the changes acceptable?** + If you want to share your work-in-progress code on a feature or bugfix, + then you can open a ``WIP``-prefixed pull request, publish patches on + the `issue tracker `_, or create a public + fork of the repository. + +* **Do the checks on the pull request show that the test suite passes?** + Make sure that all of the status checks are passing. + +* **Is the patch in a good state?** + Check :ref:`patch` and :ref:`helptriage` to review what is expected of + a patch. + +* **Does the patch break backwards-compatibility without a strong reason?** + :ref:`Run the entire test suite ` to make sure that everything + still passes. If there is a change to the semantics, then there needs to + be a strong reason, because it will cause some peoples' code to break. + If you are unsure if the breakage is worth it, then ask on python-dev. + +* **Does documentation need to be updated?** + If the pull request introduces backwards-incompatible changes (e.g. + deprecating or removing a feature), then make sure that those changes + are reflected in the documentation before you merge the pull request. + +* **Were appropriate labels added to signify necessary backporting of the pull request?** + If it is determined that a pull request needs to be + backported into one or more of the maintenance branches, then a core + developer can apply the label ``needs backport to X.Y`` to the pull + request. Once the backport pull request has been created, remove the + ``needs backport to X.Y`` label from the original pull request. (Only + core developers and members of the `Python Triage Team`_ can apply + labels to GitHub pull requests). + +* **Does the pull request have a label indicating that the submitter has signed the CLA?** + Make sure that the contributor has signed a `Contributor + Licensing Agreement `_ + (CLA), unless their change has no possible intellectual property + associated with it (e.g. fixing a spelling mistake in documentation). + To check if a contributor’s CLA has been received, go + to `Check Python CLA `_ and + put in their GitHub username. For further questions about the CLA + process, write to contributors@python.org. + +* **Were** ``What's New in Python`` **and** ``Misc/NEWS.d/next`` **updated?** + If the change is particularly interesting for end users (e.g. new features, + significant improvements, or backwards-incompatible changes), then an + entry in the ``What's New in Python`` document (in ``Doc/whatsnew/``) should + be added as well. Changes that affect only documentation generally do not + require a ``NEWS`` entry. (See the following section for more information.) + + +Updating NEWS and What's New in Python +-------------------------------------- -Always get a `Contributor Licensing Agreement`_ (CLA) signed unless the -change has no possible intellectual property associated with it (e.g. fixing -a spelling mistake in documentation). Otherwise it is simply safer from a -legal standpoint to require the contributor to sign the CLA. - -These days, the CLA can be signed electronically through the form linked -above, and this process is strongly preferred to the old mechanism that -involved sending a scanned copy of the signed paper form. - -As discussed on the PSF Contribution_ page, it is the CLA itself that gives -the PSF the necessary relicensing rights to redistribute contributions under -the Python license stack. This is an additional permission granted above and -beyond the normal permissions provided by the chosen open source license. - -Some developers may object to the relicensing permissions granted to the PSF -by the CLA. They're entirely within their rights to refuse to sign the CLA -on that basis, but that refusal *does* mean we **can't accept their patches** -for inclusion. - - -.. _Contribution: https://www.python.org/psf/contrib/ -.. _Contributor Licensing Agreement: - https://www.python.org/psf/contrib/contrib-form/ - - -Checking if the CLA has been received -------------------------------------- - -To check if a contributor's CLA has been received, go to the following website:: +Almost all changes made to the code base deserve an entry in ``Misc/NEWS.d``. +If the change is particularly interesting for end users (e.g. new features, +significant improvements, or backwards-incompatible changes), then an entry in +the ``What's New in Python`` document (in ``Doc/whatsnew/``) should be added +as well. Changes that affect documentation only generally do not require +a ``NEWS`` entry. - https://check-python-cla.herokuapp.com/ +There are two notable exceptions to this general principle, and they +both relate to changes that: -and put in their GitHub username. +* Already have a ``NEWS`` entry +* Have not yet been included in any formal release (including alpha + and beta releases) -For further questions about the CLA process, write to: contributors@python.org. +These are the two exceptions: +#. **If a change is reverted prior to release**, then the corresponding + entry is simply removed. Otherwise, a new entry must be added noting + that the change has been reverted (e.g. when a feature is released in + an alpha and then cut prior to the first beta). -What's New and News Entries ---------------------------- +#. **If a change is a fix (or other adjustment) to an earlier unreleased + change and the original** ``NEWS`` **entry remains valid**, then no additional + entry is needed. -Almost all changes made to the code base deserve an entry in ``Misc/NEWS.d``. -If the change is particularly interesting for end users (e.g. new features, -significant improvements, or backwards-incompatible changes), an entry in -the ``What's New in Python`` document (in ``Doc/whatsnew/``) should be added -as well. Changes that affect documentation only generally do not require -a news entry. +If a change needs an entry in ``What's New in Python``, then it very +likely not suitable for including in a maintenance release. -There are two notable exceptions to this general principle, and they -both relate to changes that *already* have a news entry, and have not yet -been included in any formal release (including alpha and beta releases). -These exceptions are: - -* If a change is reverted prior to release, then the corresponding entry - is simply removed. Otherwise, a new entry must be added noting that the - change has been reverted (e.g. when a feature is released in an alpha and - then cut prior to the first beta). - -* If a change is a fix (or other adjustment) to an earlier unreleased change - and the original news entry remains valid, then no additional entry is - needed. - -Needing a What's New entry almost always means that a change is *not* -suitable for inclusion in a maintenance release. A small number of -exceptions have been made for Python 2.7 due to the long support period - -when implemented, these changes *must* be noted in the "New Additions in -Python 2.7 Maintenance Releases" section of the Python 2.7 What's New -document. - -News entries go into the ``Misc/NEWS.d`` directory as individual files. The -news entry can be created by using `blurb-it `_, +``NEWS`` entries go into the ``Misc/NEWS.d`` directory as individual files. The +``NEWS`` entry can be created by using `blurb-it `_, or the `blurb `_ tool and its ``blurb add`` command. -If you are unable to use the tool you can create the news entry file manually. -The ``Misc/NEWS.d`` directory contains a sub-directory named ``next`` which -itself contains various sub-directories representing classifications for what -was affected (e.g. ``Misc/NEWS.d/next/Library`` for changes relating to the -standard library). The file name itself should be of the format +If you are unable to use the tool, then you can create the ``NEWS`` entry file +manually. The ``Misc/NEWS.d`` directory contains a sub-directory named +``next``, which contains various sub-directories representing classifications +for what was affected (e.g. ``Misc/NEWS.d/next/Library`` for changes relating +to the standard library). The file name itself should be in the format ``.bpo-..rst``: -* ```` is today's date joined with a ``-`` to the current - time, in ``YYYY-MM-DD-hh-mm-ss`` format, e.g. ``2017-05-27-16-46-23`` -* ```` is the issue number the change is for, e.g. ``12345`` - for ``bpo-12345`` -* ```` is some "unique" string to guarantee the file name is - unique across branches, e.g. ``Yl4gI2`` (typically six characters, but it can - be any length of letters and numbers, and its uniqueness can be satisfied by - typing random characters on your keyboard) +* ```` is today's date joined with a hyphen (``-``) to the current + time, in the ``YYYY-MM-DD-hh-mm-ss`` format (e.g. ``2017-05-27-16-46-23``). +* ```` is the issue number the change is for (e.g. ``12345`` + for ``bpo-12345``). +* ```` is a unique string to guarantee that the file name is + unique across branches (e.g. ``Yl4gI2``). It is typically six characters + long, but it can be any length of letters and numbers. Its uniqueness + can be satisfied by typing random characters on your keyboard. -So a file name may be +As a result, a file name can look something like ``Misc/NEWS.d/next/Library/2017-05-27-16-46-23.bpo-12345.Yl4gI2.rst``. -The contents of a news file should be valid reStructuredText. An 80 character +The contents of a ``NEWS`` file should be valid reStructuredText. An 80 character column width should be used. There is no indentation or leading marker in the file (e.g. ``-``). There is also no need to start the entry with the issue -number as it's part of the file name itself. You can use -:ref:`inline markups ` too. Example news entry:: +number since it is part of the file name. You can use +:ref:`inline markups ` too. Here is an example of a ``NEWS`` +entry:: Fix warning message when :func:`os.chdir` fails inside :func:`test.support.temp_cwd`. Patch by Chris Jerdonek. -The inline Sphinx roles like ``:func:`` can be used to assist readers in finding -more information. You can build html and verify that the link target is -appropriate by using :ref:`make html `. +The inline Sphinx roles like ``:func:`` can be used help readers +find more information. You can build HTML and verify that the +link target is appropriate by using :ref:`make html `. While Sphinx roles can be beneficial to readers, they are not required. Inline ````code blocks```` can be used instead. @@ -222,30 +157,28 @@ Working with Git_ .. seealso:: :ref:`gitbootcamp` -As a core developer, the ability to push changes to the official Python -repositories means you have to be more careful with your workflow: +As a core developer, you have the ability to push changes to the official +Python repositories, so you need to be careful with your workflow: -* You should not push new branches to the main repository. You can still use - them in your fork that you use for development of patches; you can - also push these branches to a **separate** public repository that will be - dedicated to maintenance of the work before the work gets integrated in the - main repository. +* **You should not push new branches to the main repository.** You can + still use them in the fork that you use for the development of patches. + You can also push these branches to a separate public repository + for maintenance work before it is integrated into the main repository. -* You should not commit directly into the ``master`` branch, or any of the - maintenance branches (currently ``3.9`` and ``3.8``). - You should commit against your own feature branch, and create a pull request. +* **You should not commit directly into the** ``master`` **branch, or any of the maintenance branches.** + You should commit against your own feature branch, and then create a + pull request. -* For a small change, you can make a quick edit through the GitHub web UI. +* **For a small change, you can make a quick edit through the GitHub web UI.** If you choose to use the web UI, be aware that GitHub will - create a new branch in the **main** CPython repo (not your fork). Please - delete this newly created branch after it has been merged into the + create a new branch in the main CPython repository rather than in your fork. + Delete this newly created branch after it has been merged into the ``master`` branch or any of the maintenance branches. To keep the CPython - repo tidy, please try to limit the existence of the new branch to, at most, - a few days. + repository tidy, remove the new branch within a few days. -It is recommended to keep a fork of the main repository around, as it allows -simple reversion of all local changes (even "committed" ones) if your local -clone gets into a state you aren't happy with. +Keep a fork of the main repository, since it will allow you to revert all +local changes (even committed ones) if you're not happy with your local +clone. .. _Git: https://git-scm.com/ @@ -253,65 +186,66 @@ clone gets into a state you aren't happy with. .. _committing-active-branches: -Active branches -''''''''''''''' +Seeing active branches +'''''''''''''''''''''' -If you do ``git branch`` you will see a :ref:`list of branches `. -``master`` is the in-development branch, and is the only branch that receives -new features. The other branches only receive bug fixes or security fixes. +If you use ``git branch``, then you will see a :ref:`list of branches +`. The only branch that receives new features is +``master``, the in-development branch. The other branches receive only +bug fixes or security fixes. .. _branch-merge: -Backporting Changes to an Older Version +Backporting changes to an older version ''''''''''''''''''''''''''''''''''''''' -When it is determined that a pull request needs to be backported into one or -more of the maintenance branches, a core developer can apply the labels +If it is determined that a pull request needs to be backported into one or +more of the maintenance branches, then a core developer can apply the label ``needs backport to X.Y`` to the pull request. After the pull request has been merged, miss-islington (bot) will first try to -do the backport automatically. In case that miss-islington is unable to do it, +do the backport automatically. If miss-islington is unable to do it, then the pull request author or the core developer who merged it should look into backporting it themselves, using the backport generated by cherry_picker.py_ as a starting point. -The commit hash can be obtained from the original pull request, or by using -``git log`` on the ``master`` branch. -To display the 10 most recent commit hashes and their first line of the commit -message:: +You can get the commit hash from the original pull request, or you can use +``git log`` on the ``master`` branch. To display the 10 most recent commit +hashes and their first line of the commit, use the following command:: git log -10 --oneline .. _backport-pr-title: -Prefix the backport pull request with the branch, and reference the pull request -number from ``master``, for example:: +You can prefix the backport pull request with the branch, and reference +the pull request number from ``master``. Here is an example:: [3.9] bpo-12345: Fix the Spam Module (GH-NNNN) Note that cherry_picker.py_ adds the branch prefix automatically. Once the backport pull request has been created, remove the -``needs backport to X.Y`` label from the original pull request. (Only Core -Developers and members of the `Python Triage Team`_ can apply labels to GitHub -pull requests). +``needs backport to X.Y`` label from the original pull request. (Only +core developers and members of the `Python Triage Team`_ can apply +labels to GitHub pull requests). .. _cherry_picker.py: https://github.com/python/cherry-picker .. _`Python Triage Team`: https://devguide.python.org/triaging/#python-triage-team -Reverting a Merged Pull Request +Reverting a merged pull request ''''''''''''''''''''''''''''''' -To revert a merged pull request, press the ``Revert`` button at the bottom of -the pull request. It will bring up the page to create a new pull request where -the commit can be reverted. It also creates a new branch on the main CPython -repository. Delete the branch once the pull request has been merged. +To revert a merged pull request, press the ``Revert`` button at the +bottom of the pull request. That will bring up the page to create a +new pull request where the commit can be reverted. It will also create +a new branch on the main CPython repository. Delete the branch once +the pull request has been merged. -Always include the reason for reverting the commit to help others understand -why it was done. The reason should be included as part of the commit message, -for example:: +Always include the reason for reverting the commit to help others +understand why it was done. The reason should be included as part of +the commit message. Here is an example:: Revert bpo-NNNN: Fix Spam Module (GH-111) From 41dde3c86da26be2a15e1387be18c46f3c3bbbf1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jes=C3=BAs=20Cea?= Date: Tue, 2 Feb 2021 19:14:54 +0100 Subject: [PATCH 010/793] Typo (#658) --- garbage_collector.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/garbage_collector.rst b/garbage_collector.rst index 68a4df028..0dbac59fc 100644 --- a/garbage_collector.rst +++ b/garbage_collector.rst @@ -417,7 +417,7 @@ of ``PyGC_Head`` discussed in the `Memory layout and object structure`_ section: normally assume the pointers inside the lists are in a consistent state. -* The ``_gc_prev``` field is normally used as the "previous" pointer to maintain the +* The ``_gc_prev`` field is normally used as the "previous" pointer to maintain the doubly linked list but its lowest two bits are used to keep the flags ``PREV_MASK_COLLECTING`` and ``_PyGC_PREV_MASK_FINALIZED``. Between collections, the only flag that can be present is ``_PyGC_PREV_MASK_FINALIZED`` that indicates From 6bcef3134d63dabc381f0a5a91e31bfdd5f63748 Mon Sep 17 00:00:00 2001 From: Brandt Bucher Date: Mon, 8 Feb 2021 13:55:21 -0800 Subject: [PATCH 011/793] Add myself as a pattern matching expert (#659) --- experts.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/experts.rst b/experts.rst index 473e9f997..619016d1d 100644 --- a/experts.rst +++ b/experts.rst @@ -339,6 +339,7 @@ memoryview networking giampaolo.rodola, object model benjamin.peterson, twouters packaging tarek, lemburg, alexis, eric.araujo, dstufft, paul.moore +pattern matching brandtbucher* peg parser gvanrossum, pablogsal, lys.nikolaou performance brett.cannon, vstinner, serhiy.storchaka, yselivanov, rhettinger, Mark.Shannon pip ncoghlan, dstufft, paul.moore, Marcus.Smith, pradyunsg From 23fb8b53bfac0cf271439441ab1d2a87f492e58b Mon Sep 17 00:00:00 2001 From: Andrew Kuchling Date: Mon, 22 Feb 2021 10:38:05 -0500 Subject: [PATCH 012/793] Note removal of privileges for akuchling (#660) Co-authored-by: Andrew Kuchling --- developers.csv | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/developers.csv b/developers.csv index 48cc31309..ffee4c300 100644 --- a/developers.csv +++ b/developers.csv @@ -163,7 +163,7 @@ Eric S. Raymond,,2000-06-02,2017-02-10,Did not make GitHub transition Greg Stein,,1999-11-07,2017-02-10,Did not make GitHub transition Just van Rossum,,1999-01-22,2017-02-10,Did not make GitHub transition Greg Ward,,1998-12-18,2017-02-10,Did not make GitHub transition -Andrew Kuchling,akuchling,1998-04-09,, +Andrew Kuchling,akuchling,1998-04-09,2021-02-22,Privileges relinquished on 2021-02-22 Ken Manheimer,,1998-03-03,2005-04-08,Privileges relinquished on 2005-04-08 Jeremy Hylton,jeremyhylton,1997-08-13,, Roger E. Masse,,1996-12-09,2017-02-10,Did not make GitHub transition From 29ae8afe247e9eeebba2cad66e2cfc8f9ec3fae2 Mon Sep 17 00:00:00 2001 From: Mariatta Wijaya Date: Wed, 10 Mar 2021 18:19:22 -0800 Subject: [PATCH 013/793] Updated Devcycle doc with Ee's name. (#663) cc @ewdurbin. --- devcycle.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/devcycle.rst b/devcycle.rst index 06c7eaddd..366636389 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -255,7 +255,7 @@ This role is paramount to the security of the Python Language, Community, and Infrastructure. The Executive Director of the Python Software Foundation delegates authority on -GitHub Organization Owner Status to Ernest W. Durbin III - Python Software +GitHub Organization Owner Status to Ee W. Durbin III - Python Software Foundation Director of Infrastructure. Common reasons for this role are: Infrastructure Staff Membership, Python Software Foundation General Counsel, and Python Software Foundation Staff as fallback. @@ -280,7 +280,7 @@ Current Owners +----------------------+--------------------------------+-----------------+ | Ewa Jodlowska | PSF Executive Director | ejodlowska | +----------------------+--------------------------------+-----------------+ -| Ernest W. Durbin III | PSF Director of Infrastructure | ewdurbin | +| Ee W. Durbin III | PSF Director of Infrastructure | ewdurbin | +----------------------+--------------------------------+-----------------+ | Van Lindberg | PSF General Counsel | VanL | +----------------------+--------------------------------+-----------------+ From 0f2ea524684c428b81b41517d1d150b0e6959509 Mon Sep 17 00:00:00 2001 From: Eric Snow Date: Fri, 19 Mar 2021 12:00:10 -0600 Subject: [PATCH 014/793] Fix the pyperformance Repo URL. (#664) The repo changed name, from "performance" to "pyperformance", a while back. The devguide did not get updated at that time. Updating the links here means GitHub will no longer need to rewrite the old URL in requests. --- communication.rst | 2 +- runtests.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/communication.rst b/communication.rst index 11c4c5aeb..c2fe9a7a0 100644 --- a/communication.rst +++ b/communication.rst @@ -130,4 +130,4 @@ source of benchmarks for all Python implementations. .. _Python Core Workflow: https://github.com/python/core-workflow .. _cherry_picker: https://pypi.org/project/cherry-picker/ .. _blurb: https://pypi.org/project/blurb -.. _Performance Benchmark: https://github.com/python/performance +.. _Performance Benchmark: https://github.com/python/pyperformance diff --git a/runtests.rst b/runtests.rst index ef410bdd2..74e38ff8b 100644 --- a/runtests.rst +++ b/runtests.rst @@ -129,7 +129,7 @@ Benchmarks ---------- Benchmarking is useful to test that a change does not degrade performance. -`The Python Benchmark Suite `_ +`The Python Benchmark Suite `_ has a collection of benchmarks for all Python implementations. Documentation about running the benchmarks is in the `README.txt -`_ of the repo. +`_ of the repo. From b9f1ae8c60c62d1e709659f016e73b94e0cd4d41 Mon Sep 17 00:00:00 2001 From: Antoine Pitrou Date: Sun, 21 Mar 2021 18:06:23 +0100 Subject: [PATCH 015/793] Fix link to custom builders (GH-666) --- buildbots.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/buildbots.rst b/buildbots.rst index ce1f16845..92662487c 100644 --- a/buildbots.rst +++ b/buildbots.rst @@ -209,7 +209,7 @@ Custom builders When working on a platform-specific issue, you may want to test your changes on the buildbot fleet rather than just on Travis and AppVeyor. To do so, you can make use of the `custom builders -`_. +`_. These builders track the ``buildbot-custom`` short-lived branch of the ``python/cpython`` repository, which is only accessible to core developers. From 73e53b6ddb7152bf9fd74658b3fb802bbf87c502 Mon Sep 17 00:00:00 2001 From: "Eric V. Smith" Date: Thu, 15 Apr 2021 15:18:41 -0400 Subject: [PATCH 016/793] Allow assigning dataclasses issues to eric.smith. (#675) --- experts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experts.rst b/experts.rst index 619016d1d..1f0c96192 100644 --- a/experts.rst +++ b/experts.rst @@ -93,7 +93,7 @@ csv skip.montanaro (inactive) ctypes theller (inactive), belopolsky, amaury.forgeotdarc, meador.inge curses twouters -dataclasses eric.smith +dataclasses eric.smith* datetime belopolsky, p-ganssle dbm decimal facundobatista, rhettinger, mark.dickinson From 6127a2b84dea97d9ee046f9d40517d2ba06bc399 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Fri, 23 Apr 2021 20:02:51 +0200 Subject: [PATCH 017/793] devcycle: Clarify doc & test changes in Beta & RC (GH-676) --- devcycle.rst | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/devcycle.rst b/devcycle.rst index 366636389..aba81d71e 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -185,9 +185,10 @@ Beta ---- After a first beta release is published, no new features are accepted. Only -bug fixes can now be committed. This is when core developers should concentrate -on the task of fixing regressions and other new issues filed by users who have -downloaded the alpha and beta releases. +bug fixes and improvements to documentation and tests can now be committed. +This is when core developers should concentrate on the task of fixing +regressions and other new issues filed by users who have downloaded the alpha +and beta releases. Being in beta can be viewed much like being in RC_ but without the extra overhead of needing commit reviews. @@ -207,6 +208,10 @@ severe enough (e.g. crashes) that they deserve fixing before the final release. All other issues should be deferred to the next development cycle, since stability is the strongest concern at this point. +While the goal is to have no code changes between a RC and a final release, +there may be a need for final documentation or test fixes. Any such proposed +changes should be discussed first with the release manager. + You **cannot** skip the peer review during an RC, no matter how small! Even if it is a simple copy-and-paste change, **everything** requires peer review from a core developer. From 206e583c3defb77ff13b320a84a1bf487def51bc Mon Sep 17 00:00:00 2001 From: Mariatta Wijaya Date: Thu, 29 Apr 2021 12:56:37 -0700 Subject: [PATCH 018/793] Add GitHub Actions to run the CI (#680) set continue-on-error true on the linkcheck step --- .github/workflows/ci.yml | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 .github/workflows/ci.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 000000000..09cd927e7 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,35 @@ +name: Tests + +on: + pull_request: + push: + branches: master + +jobs: + test: + name: test w/ Python ${{ matrix.python-version }} + + runs-on: ubuntu-latest + + strategy: + matrix: + python-version: ["3.9"] + + steps: + - uses: actions/checkout@v2 + - uses: actions/cache@v1 + with: + path: ~/.cache/pip + key: ${{ runner.os }}-pip-${{ hashFiles('pyproject.toml') }} + restore-keys: | + ${{ runner.os }}-pip- + - uses: actions/setup-python@v2 + with: + python-version: ${{ matrix.python-version }} + - name: Install Dependencies + run: python3 -m pip install -U pip -r requirements.txt + - name: Build Docs + run: sphinx-build -n -W -q -b html -d _build/doctrees . _build/html + - name: Link Check + run: sphinx-build -b linkcheck -d _build/doctrees . _build/linkcheck + continue-on-error: true From fb051b71da938c95d52880e0f964a84db67eb2e1 Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Thu, 29 Apr 2021 22:05:01 -0400 Subject: [PATCH 019/793] Remove mention of old coverage comment from checklist (#685) --- committing.rst | 4 ---- 1 file changed, 4 deletions(-) diff --git a/committing.rst b/committing.rst index a242e8aa9..4cb92e23d 100644 --- a/committing.rst +++ b/committing.rst @@ -23,10 +23,6 @@ to enter the public source tree. Ask yourself the following questions: in-development branch. Pull requests should only target bug-fix branches if an issue appears in only that version and possibly older versions. -* **Are there comments on the pull request?** - Look for explanations about whether the code coverage increased or - stayed the same. - * **Are the changes acceptable?** If you want to share your work-in-progress code on a feature or bugfix, then you can open a ``WIP``-prefixed pull request, publish patches on From 0af1395fbe184ab48b9b154054fbe07765965356 Mon Sep 17 00:00:00 2001 From: "dependabot-preview[bot]" <27856297+dependabot-preview[bot]@users.noreply.github.com> Date: Thu, 29 Apr 2021 19:05:29 -0700 Subject: [PATCH 020/793] Upgrade to GitHub-native Dependabot (#684) Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com> --- .github/dependabot.yml | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 .github/dependabot.yml diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 000000000..491deae0a --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,7 @@ +version: 2 +updates: +- package-ecosystem: pip + directory: "/" + schedule: + interval: daily + open-pull-requests-limit: 10 From 5048d2e5abbee689c2462d73c93e00fb8ccf6fc0 Mon Sep 17 00:00:00 2001 From: Himanshu Date: Mon, 3 May 2021 03:05:45 +0530 Subject: [PATCH 021/793] Fix broken links (#683) --- buildworker.rst | 2 +- communication.rst | 4 ++-- documenting.rst | 4 ++-- help.rst | 2 +- index.rst | 2 +- motivations.rst | 2 +- 6 files changed, 8 insertions(+), 8 deletions(-) diff --git a/buildworker.rst b/buildworker.rst index 3e5deabe1..99f758179 100644 --- a/buildworker.rst +++ b/buildworker.rst @@ -198,7 +198,7 @@ Latent workers ^^^^^^^^^^^^^^ We also support running `latent workers -`_ +`_ on the AWS EC2 service. To set up such a worker: * Start an instance of your chosen base AMI and set it up as a diff --git a/communication.rst b/communication.rst index c2fe9a7a0..90b5a1027 100644 --- a/communication.rst +++ b/communication.rst @@ -56,8 +56,8 @@ on Freenode_. mailing list is the place to discuss and work on improvements to the CPython core development workflow. -A complete list of Python mailing lists can be found at https://mail.python.org. -Most lists are also mirrored at `GMANE `_ and can be read and +A complete list of Python mailing lists can be found at https://mail.python.org/mailman/listinfo. +Most lists are also mirrored at `GMANE `_ and can be read and posted to in various ways, including via web browsers, NNTP newsreaders, and RSS feed readers. diff --git a/documenting.rst b/documenting.rst index ebf5a58e2..5c712a001 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1679,8 +1679,8 @@ in production, other are work in progress: .. _mail_it: https://mail.python.org/pipermail/doc-sig/2019-April/004114.html .. _mail_lt: https://mail.python.org/pipermail/doc-sig/2019-July/004138.html .. _mail_ru: https://mail.python.org/pipermail/doc-sig/2019-May/004131.html -.. _tx_pl: https://www.transifex.com/python-doc/python-newest/language/pl/ -.. _tx_zh_cn: https://www.transifex.com/python-doc/python-newest/language/ +.. _tx_pl: https://www.transifex.com/python-doc/python-newest/ +.. _tx_zh_cn: https://www.transifex.com/python-doc/python-newest/ .. _tx_zh_tw: https://www.transifex.com/python-tw-doc/python-36-tw .. _wiki_pt_br: http://python.org.br/traducao diff --git a/help.rst b/help.rst index 50e3b982a..b0c726cdc 100644 --- a/help.rst +++ b/help.rst @@ -107,7 +107,7 @@ during office hours. | | | 24 hours before the start. | +------------------+-------------------------------+------------------------------------------------+ -.. _Python's Zulip Chat: https://python.zulipchat.com/#narrow/stream/116503-core/topic/Office.20Hour +.. _Python's Zulip Chat: https://python.zulipchat.com/login/#narrow/stream/116503-core/topic/Office.20Hour .. _Mariatta's twitter: https://twitter.com/mariatta diff --git a/index.rst b/index.rst index f62a7eef8..90cc5de0c 100644 --- a/index.rst +++ b/index.rst @@ -335,7 +335,7 @@ Full Table of Contents .. _python.org maintenance: https://pythondotorg.readthedocs.io/ .. _Python: https://www.python.org/ .. _Core Python Mentorship: https://www.python.org/dev/core-mentorship/ -.. _PyPy: http://www.pypy.org/ +.. _PyPy: http://www.pypy.org .. _Jython: http://www.jython.org/ .. _IronPython: http://ironpython.net/ .. _Stackless: http://www.stackless.com/ diff --git a/motivations.rst b/motivations.rst index 50085000c..f1a1a41cc 100644 --- a/motivations.rst +++ b/motivations.rst @@ -118,7 +118,7 @@ participating in the CPython core development process: * Microsoft (Software Developer) * Personal site: `stevedower.id.au `_ - * Speaking: `stevedower.id.au/speaking `_ + * Speaking: `stevedower.id.au/speaking `_ * Work blog: `aka.ms/pythonblog `_ * Email address: steve.dower@python.org From acc834ccc95364706c9cdc95553d63f3320cda65 Mon Sep 17 00:00:00 2001 From: Mariatta Wijaya Date: Mon, 3 May 2021 14:39:18 -0700 Subject: [PATCH 022/793] Replace CPython default branch name to main branch. (#662) --- .github/CONTRIBUTING.md | 2 +- buildbots.rst | 2 +- committing.rst | 12 ++++----- coverity.rst | 2 +- devcycle.rst | 4 +-- gitbootcamp.rst | 59 +++++++++++++++++++++++++---------------- index.rst | 10 +++---- pullrequest.rst | 8 +++--- setup.rst | 2 +- triaging.rst | 54 ++++++++++++++++++------------------- 10 files changed, 84 insertions(+), 71 deletions(-) diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md index 87686d966..4b29950db 100644 --- a/.github/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -24,7 +24,7 @@ our workflow that are not covered by a bot or status check are: - All discussions that are not directly related to the code in the pull request should happen on [bugs.python.org](https://bugs.python.org/) - Upon your first non-trivial pull request (which includes documentation changes), - feel free to add yourself to [`Misc/ACKS`](https://github.com/python/cpython/blob/master/Misc/ACKS) + feel free to add yourself to [`Misc/ACKS`](https://github.com/python/cpython/blob/main/Misc/ACKS) ## Setting Expectations diff --git a/buildbots.rst b/buildbots.rst index 92662487c..58535ca1e 100644 --- a/buildbots.rst +++ b/buildbots.rst @@ -40,7 +40,7 @@ There are three ways of visualizing recent build results: https://code.google.com/archive/p/bbreport. Installing it is trivial: just add the directory containing ``bbreport.py`` to your system path so that you can run it from any filesystem location. For example, if you want - to display the latest build results on the development ("master") branch, + to display the latest build results on the development ("main") branch, type:: bbreport.py -q 3.x diff --git a/committing.rst b/committing.rst index 4cb92e23d..213a2292e 100644 --- a/committing.rst +++ b/committing.rst @@ -19,7 +19,7 @@ to enter the public source tree. Ask yourself the following questions: we need to have a resolution there before we can merge the pull request. * **Was the pull request first made against the appropriate branch?** - The only branch that receives new features is ``master``, the + The only branch that receives new features is ``main``, the in-development branch. Pull requests should only target bug-fix branches if an issue appears in only that version and possibly older versions. @@ -161,7 +161,7 @@ Python repositories, so you need to be careful with your workflow: You can also push these branches to a separate public repository for maintenance work before it is integrated into the main repository. -* **You should not commit directly into the** ``master`` **branch, or any of the maintenance branches.** +* **You should not commit directly into the** ``main`` **branch, or any of the maintenance branches.** You should commit against your own feature branch, and then create a pull request. @@ -169,7 +169,7 @@ Python repositories, so you need to be careful with your workflow: If you choose to use the web UI, be aware that GitHub will create a new branch in the main CPython repository rather than in your fork. Delete this newly created branch after it has been merged into the - ``master`` branch or any of the maintenance branches. To keep the CPython + ``main`` branch or any of the maintenance branches. To keep the CPython repository tidy, remove the new branch within a few days. Keep a fork of the main repository, since it will allow you to revert all @@ -187,7 +187,7 @@ Seeing active branches If you use ``git branch``, then you will see a :ref:`list of branches `. The only branch that receives new features is -``master``, the in-development branch. The other branches receive only +``main``, the in-development branch. The other branches receive only bug fixes or security fixes. @@ -207,7 +207,7 @@ backporting it themselves, using the backport generated by cherry_picker.py_ as a starting point. You can get the commit hash from the original pull request, or you can use -``git log`` on the ``master`` branch. To display the 10 most recent commit +``git log`` on the ``main`` branch. To display the 10 most recent commit hashes and their first line of the commit, use the following command:: git log -10 --oneline @@ -215,7 +215,7 @@ hashes and their first line of the commit, use the following command:: .. _backport-pr-title: You can prefix the backport pull request with the branch, and reference -the pull request number from ``master``. Here is an example:: +the pull request number from ``main``. Here is an example:: [3.9] bpo-12345: Fix the Spam Module (GH-NNNN) diff --git a/coverity.rst b/coverity.rst index 8de943aef..2a9359715 100644 --- a/coverity.rst +++ b/coverity.rst @@ -150,4 +150,4 @@ Dakshesh Vyas .. _Coverity Connect: https://scan.coverity.com/projects/python -.. _coverity_model.c: https://github.com/python/cpython/blob/master/Misc/coverity_model.c +.. _coverity_model.c: https://github.com/python/cpython/blob/main/Misc/coverity_model.c diff --git a/devcycle.rst b/devcycle.rst index aba81d71e..e8bb60934 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -44,7 +44,7 @@ There is a branch for each *feature version*, whether released or not (e.g. In-development (main) branch ---------------------------- -The ``master`` branch is the branch for the next feature release; it is +The ``main`` branch is the branch for the next feature release; it is under active development for all kinds of changes: new features, semantic changes, performance improvements, bug fixes. @@ -57,7 +57,7 @@ release was cut (for example, 3.4.0 final). Starting with the 3.5 release, we create the release maintenance branch (e.g. 3.5) at the time we enter beta (3.5.0 beta 1). This allows -feature development for the release 3.n+1 to occur within the master +feature development for the release 3.n+1 to occur within the main branch alongside the beta and release candidate stabilization periods for release 3.n. diff --git a/gitbootcamp.rst b/gitbootcamp.rst index bd5f8e441..9cb76ac2f 100644 --- a/gitbootcamp.rst +++ b/gitbootcamp.rst @@ -103,17 +103,17 @@ Creating and Switching Branches ------------------------------- .. important:: - Never commit directly to the ``master`` branch. + Never commit directly to the ``main`` branch. Create a new branch and switch to it:: - # creates a new branch off master and switch to it - git checkout -b master + # creates a new branch off main and switch to it + git checkout -b main This is equivalent to:: - # create a new branch from master, without checking it out - git branch master + # create a new branch from main, without checking it out + git branch main # check out the branch git checkout @@ -144,7 +144,7 @@ Deleting Branches To delete a **local** branch that you no longer need:: - git checkout master + git checkout main git branch -D To delete a **remote** branch:: @@ -153,6 +153,19 @@ To delete a **remote** branch:: You may specify more than one branch for deletion. + +Renaming Branch +--------------- + +The CPython repository's default branch was renamed from ``master`` to ``main`` +after the Python 3.10b1 release. If you had cloned the repository before this +change, you can rename your local branch as follows:: + + git branch -m master main + git fetch upstream + git branch -u upstream/main main + + Staging and Committing Files ---------------------------- @@ -230,7 +243,7 @@ Creating a Pull Request 3. Click the ``compare across forks`` link. -4. Select the base repository: ``python/cpython`` and base branch: ``master``. +4. Select the base repository: ``python/cpython`` and base branch: ``main``. 5. Select the head repository: ``/cpython`` and head branch: the branch containing your changes. @@ -250,14 +263,14 @@ Scenario: the upstream CPython repository. Please do not try to solve this by creating a pull request from -``python:master`` to ``:master`` as the authors of the patches will +``python:main`` to ``:main`` as the authors of the patches will get notified unnecessarily. Solution:: - git checkout master - git pull upstream master - git push origin master + git checkout main + git pull upstream main + git push origin main .. note:: For the above commands to work, please follow the instructions found in the :ref:`checkout` section @@ -275,11 +288,11 @@ Solution:: git checkout some-branch git fetch upstream - git merge upstream/master + git merge upstream/main git push origin some-branch You may see error messages like "CONFLICT" and "Automatic merge failed;" when -you run ``git merge upstream/master``. +you run ``git merge upstream/main``. When it happens, you need to resolve conflict. See these articles about resolving conflicts: @@ -308,7 +321,7 @@ Solution: .. code-block:: bash - git checkout $(git rev-list -n 1 --before="yyyy-mm-dd hh:mm:ss" master) + git checkout $(git rev-list -n 1 --before="yyyy-mm-dd hh:mm:ss" main) git apply /path/to/issueNNNN-git.patch If the patch still won't apply, then a patch tool will not be able to @@ -321,7 +334,7 @@ Solution: 5. If the patch was applied to an old revision, it needs to be updated and merge conflicts need to be resolved:: - git rebase master + git rebase main git mergetool 6. Push the changes and open a pull request. @@ -388,7 +401,7 @@ Pull requests can be accepted and merged by a Python Core Developer. bpo-12345: Improve the spam module (#777) * Improve the spam module - * merge from master + * merge from main * adjust code based on review comment * rebased @@ -402,7 +415,7 @@ Backporting Merged Changes -------------------------- A pull request may need to be backported into one of the maintenance branches -after it has been accepted and merged into ``master``. It is usually indicated +after it has been accepted and merged into ``main``. It is usually indicated by the label ``needs backport to X.Y`` on the pull request itself. Use the utility script @@ -411,10 +424,10 @@ from the `core-workflow `_ repository to backport the commit. The commit hash for backporting is the squashed commit that was merged to -the ``master`` branch. On the merged pull request, scroll to the bottom of the +the ``main`` branch. On the merged pull request, scroll to the bottom of the page. Find the event that says something like:: - merged commit into python:master ago. + merged commit into python:main ago. By following the link to ````, you will get the full commit hash. @@ -459,12 +472,12 @@ items like updating ``Misc/ACKS``. .. _Allow edits from maintainers: https://help.github.com/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork/ -To edit an open pull request that targets ``master``: +To edit an open pull request that targets ``main``: 1. In the pull request page, under the description, there is some information about the contributor's forked CPython repository and branch name that will be useful later:: - wants to merge 1 commit into python:master from : + wants to merge 1 commit into python:main from : 2. Fetch the pull request, using the :ref:`git pr ` alias:: @@ -473,13 +486,13 @@ To edit an open pull request that targets ``master``: This will checkout the contributor's branch at ````. 3. Make and commit your changes on the branch. For example, merge in changes - made to ``master`` since the PR was submitted (any merge commits will be + made to ``main`` since the PR was submitted (any merge commits will be removed by the later ``Squash and Merge`` when accepting the change): .. code-block:: bash git fetch upstream - git merge upstream/master + git merge upstream/main git add git commit -m "" diff --git a/index.rst b/index.rst index 90cc5de0c..7da47b14e 100644 --- a/index.rst +++ b/index.rst @@ -52,7 +52,7 @@ instructions please see the :ref:`setup guide `. 5. Create a new branch where your work for the issue will go, e.g.:: - git checkout -b fix-issue-12345 master + git checkout -b fix-issue-12345 main If an issue does not already exist, please `create it `_. Trivial issues (e.g. typo fixes) do not @@ -98,7 +98,7 @@ Status of Python branches +------------------+--------------+-------------+----------------+----------------+-----------------------+ | Branch | Schedule | Status | First release | End-of-life | Release manager | +==================+==============+=============+================+================+=======================+ -| master | :pep:`619` | features | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +| main | :pep:`619` | features | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.9 | :pep:`596` | bugfix | 2020-10-05 | *TBD* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ @@ -111,7 +111,7 @@ Status of Python branches .. Remember to update the end-of-life table in devcycle.rst. -The master branch is currently the future Python 3.10, and is the only +The main branch is currently the future Python 3.10, and is the only branch that accepts new features. The latest release for each Python version can be found on the `download page `_. @@ -244,7 +244,7 @@ Key Resources * `Buildbot status`_ * Source code * `Browse online `_ - * `Snapshot of the *master* branch `_ + * `Snapshot of the *main* branch `_ * `Daily OS X installer `_ * PEPs_ (Python Enhancement Proposals) * :doc:`help` @@ -330,7 +330,7 @@ Full Table of Contents appendix .. _Buildbot status: https://www.python.org/dev/buildbot/ -.. _Misc directory: https://github.com/python/cpython/tree/master/Misc +.. _Misc directory: https://github.com/python/cpython/tree/main/Misc .. _PEPs: https://www.python.org/dev/peps/ .. _python.org maintenance: https://pythondotorg.readthedocs.io/ .. _Python: https://www.python.org/ diff --git a/pullrequest.rst b/pullrequest.rst index ef85923c0..7cb194490 100644 --- a/pullrequest.rst +++ b/pullrequest.rst @@ -75,7 +75,7 @@ You should have already :ref:`set up your system `, * Create a new branch in your local clone:: - git checkout -b upstream/master + git checkout -b upstream/main * Make changes to the code, and use ``git status`` and ``git diff`` to see them. @@ -122,12 +122,12 @@ You should have already :ref:`set up your system `, there are merge conflicts, git will warn you about this and enter conflict resolution mode. See :ref:`resolving-merge-conflicts` below. -* If time passes and there are merge conflicts with the master branch, GitHub +* If time passes and there are merge conflicts with the main branch, GitHub will show a warning to this end and you may be asked to address this. Merge - the changes from the master branch while resolving the conflicts locally:: + the changes from the main branch while resolving the conflicts locally:: git checkout - git pull upstream master # pull = fetch + merge + git pull upstream main # pull = fetch + merge # resolve conflicts: see "Resolving Merge Conflicts" below git push origin diff --git a/setup.rst b/setup.rst index 0b5e86441..385f142f2 100644 --- a/setup.rst +++ b/setup.rst @@ -280,7 +280,7 @@ to build. .. _this documentation: https://cpython-core-tutorial.readthedocs.io/en/latest/build_cpython_windows.html .. _Visual Studio 2017: https://www.visualstudio.com/ -.. _readme: https://github.com/python/cpython/blob/master/PCbuild/readme.txt +.. _readme: https://github.com/python/cpython/blob/main/PCbuild/readme.txt .. _PCbuild directory: https://github.com/python/cpython/tree/2.7/PCbuild/ .. _2.7 readme: https://github.com/python/cpython/blob/2.7/PCbuild/readme.txt .. _PC/VS9.0 directory: https://github.com/python/cpython/tree/2.7/PC/VS9.0/ diff --git a/triaging.rst b/triaging.rst index a7bfec465..60025394b 100644 --- a/triaging.rst +++ b/triaging.rst @@ -103,13 +103,13 @@ expert-asyncio invalid Used manually for PRs that do not meet basic requirements and automatically added by bedevere when PR authors attempt to merge maintenace - branches into the master branch. During events such as the October + branches into the main branch. During events such as the October Hacktoberfest, this label will prevent the PR from counting toward the author's contributions. needs backport to X.Y Used for PRs which are appropriate to backport to - branches prior to master. Generally, backports to the maintenance branches + branches prior to main. Generally, backports to the maintenance branches are primarily bugfixes and documentation clarifications. Backports to the security branches are strictly reserved for PRs involving security fixes, such as crashes, privilege escalation, and DoS. The use of this label will cause @@ -582,31 +582,31 @@ Checklist for Triaging .. _CPython: https://github.com/python/cpython/ -.. _Doc: https://github.com/python/cpython/tree/master/Doc/ -.. _Grammar: https://github.com/python/cpython/tree/master/Grammar/ -.. _Lib: https://github.com/python/cpython/tree/master/Lib/ -.. _Lib/lib2to3: https://github.com/python/cpython/tree/master/Lib/lib2to3/ -.. _Lib/ctypes: https://github.com/python/cpython/tree/master/Lib/ctypes/ -.. _Lib/distutils: https://github.com/python/cpython/tree/master/Lib/distutils/ -.. _Lib/doctest.py: https://github.com/python/cpython/tree/master/Lib/doctest.py -.. _Lib/idlelib: https://github.com/python/cpython/tree/master/Lib/idlelib/ -.. _Lib/io.py: https://github.com/python/cpython/tree/master/Lib/io.py -.. _Lib/re.py: https://github.com/python/cpython/tree/master/Lib/re.py -.. _Lib/test: https://github.com/python/cpython/tree/master/Lib/test/ -.. _Lib/test/regrtest.py: https://github.com/python/cpython/tree/master/Lib/test/regrtest.py -.. _Lib/test/support: https://github.com/python/cpython/tree/master/Lib/test/support/ -.. _Lib/tkinter: https://github.com/python/cpython/tree/master/Lib/tkinter/ -.. _Lib/unittest: https://github.com/python/cpython/tree/master/Lib/unittest/ -.. _Lib/xml: https://github.com/python/cpython/tree/master/Lib/xml/ -.. _Modules: https://github.com/python/cpython/tree/master/Modules/ -.. _Modules/_io: https://github.com/python/cpython/tree/master/Modules/_io/ -.. _Modules/_sre.c: https://github.com/python/cpython/tree/master/Modules/_sre.c -.. _Objects: https://github.com/python/cpython/tree/master/Objects/ -.. _Objects/unicodeobject.c: https://github.com/python/cpython/tree/master/Objects/unicodeobject.c -.. _Parser: https://github.com/python/cpython/tree/master/Parser/ -.. _Python: https://github.com/python/cpython/tree/master/Python/ -.. _Tools: https://github.com/python/cpython/tree/master/Tools/ -.. _Tools/demo: https://github.com/python/cpython/tree/master/Tools/demo/ +.. _Doc: https://github.com/python/cpython/tree/main/Doc/ +.. _Grammar: https://github.com/python/cpython/tree/main/Grammar/ +.. _Lib: https://github.com/python/cpython/tree/main/Lib/ +.. _Lib/lib2to3: https://github.com/python/cpython/tree/main/Lib/lib2to3/ +.. _Lib/ctypes: https://github.com/python/cpython/tree/main/Lib/ctypes/ +.. _Lib/distutils: https://github.com/python/cpython/tree/main/Lib/distutils/ +.. _Lib/doctest.py: https://github.com/python/cpython/tree/main/Lib/doctest.py +.. _Lib/idlelib: https://github.com/python/cpython/tree/main/Lib/idlelib/ +.. _Lib/io.py: https://github.com/python/cpython/tree/main/Lib/io.py +.. _Lib/re.py: https://github.com/python/cpython/tree/main/Lib/re.py +.. _Lib/test: https://github.com/python/cpython/tree/main/Lib/test/ +.. _Lib/test/regrtest.py: https://github.com/python/cpython/tree/main/Lib/test/regrtest.py +.. _Lib/test/support: https://github.com/python/cpython/tree/main/Lib/test/support/ +.. _Lib/tkinter: https://github.com/python/cpython/tree/main/Lib/tkinter/ +.. _Lib/unittest: https://github.com/python/cpython/tree/main/Lib/unittest/ +.. _Lib/xml: https://github.com/python/cpython/tree/main/Lib/xml/ +.. _Modules: https://github.com/python/cpython/tree/main/Modules/ +.. _Modules/_io: https://github.com/python/cpython/tree/main/Modules/_io/ +.. _Modules/_sre.c: https://github.com/python/cpython/tree/main/Modules/_sre.c +.. _Objects: https://github.com/python/cpython/tree/main/Objects/ +.. _Objects/unicodeobject.c: https://github.com/python/cpython/tree/main/Objects/unicodeobject.c +.. _Parser: https://github.com/python/cpython/tree/main/Parser/ +.. _Python: https://github.com/python/cpython/tree/main/Python/ +.. _Tools: https://github.com/python/cpython/tree/main/Tools/ +.. _Tools/demo: https://github.com/python/cpython/tree/main/Tools/demo/ .. _Developer's guide: https://github.com/python/devguide/ .. _GSoC: https://summerofcode.withgoogle.com/ .. _issue tracker: https://bugs.python.org From 534b0ec04ad448cac3384a128ff9f61e4ba0c99e Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Mon, 3 May 2021 19:14:47 -0400 Subject: [PATCH 023/793] update branch status as of 3.10.0b1 (#686) --- index.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/index.rst b/index.rst index 7da47b14e..31aa398d4 100644 --- a/index.rst +++ b/index.rst @@ -98,11 +98,13 @@ Status of Python branches +------------------+--------------+-------------+----------------+----------------+-----------------------+ | Branch | Schedule | Status | First release | End-of-life | Release manager | +==================+==============+=============+================+================+=======================+ -| main | :pep:`619` | features | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +| main | *TBD* | features | *TBD* | *TBD* | Pablo Galindo Salgado | ++------------------+--------------+-------------+----------------+----------------+-----------------------+ +| 3.10 | :pep:`619` | prerelase | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.9 | :pep:`596` | bugfix | 2020-10-05 | *TBD* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ -| 3.8 | :pep:`569` | bugfix | 2019-10-14 | *2024-10* | Łukasz Langa | +| 3.8 | :pep:`569` | security | 2019-10-14 | *2024-10* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.7 | :pep:`537` | security | 2018-06-27 | *2023-06-27* | Ned Deily | +------------------+--------------+-------------+----------------+----------------+-----------------------+ @@ -111,7 +113,7 @@ Status of Python branches .. Remember to update the end-of-life table in devcycle.rst. -The main branch is currently the future Python 3.10, and is the only +The main branch is currently the future Python 3.11, and is the only branch that accepts new features. The latest release for each Python version can be found on the `download page `_. From 9b6b939e7170c0c56cc30a9718b7a28295a53cc3 Mon Sep 17 00:00:00 2001 From: Petr Viktorin Date: Tue, 4 May 2021 17:20:57 +0200 Subject: [PATCH 024/793] Add a page on changing the C API (PEP-652) (GH-682) This repeats points currently said in `Include/README.rst`. I intend to replace most of that README with a link to this document. The main change from that document are: - The three "rings" are not defined by where things are declared. It would be good to have them in the right place, but it's not yet the case, and for technical reasons we might never reach the ideal. (In `Include/README.rst` it makes more sense to make the locations more prominent.) - The order is switched. For changing the Limited API you generally need to read (and follow guidelines in) the preceding sections as well. On top of that, I addes points from PEP 652 (Maintaining the Stable ABI). Co-Authored-By: Erlend Egeberg Aasland --- c-api.rst | 194 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ index.rst | 1 + 2 files changed, 195 insertions(+) create mode 100644 c-api.rst diff --git a/c-api.rst b/c-api.rst new file mode 100644 index 000000000..9198c80cb --- /dev/null +++ b/c-api.rst @@ -0,0 +1,194 @@ +======================= +Changing Python's C API +======================= + +The C API is divided into three sections: + +1. The internal, private API, available with ``Py_BUILD_CORE`` defined. + Ideally declared in ``Include/internal/``. Any API named with a leading + underscore is also considered private. +2. The public C API, available when ``Python.h`` is included normally. + Ideally declared in ``Include/cpython/``. +3. The Limited API, available with ``Py_LIMITED_API`` defined. + Ideally declared directly under ``Include/``. + +Each section has higher stability & maintenance requirements, and you will +need to think about more issues when you add or change definitions in it. + +The compatibility guarantees for public C API are explained in the +user documentation, ``Doc/c-api/stable.rst`` (:ref:`python:stable`). + + +The internal API +================ + +Internal API is defined in ``Include/internal/`` and is only available +for building CPython itself, as indicated by a macro like ``Py_BUILD_CORE``. + +While internal API can be changed at any time, it's still good to keep it +stable: other API or other CPython developers may depend on it. + +With PyAPI_FUNC or PyAPI_DATA +----------------------------- + +Functions or structures in ``Include/internal/`` defined with +``PyAPI_FUNC`` or ``PyAPI_DATA`` are internal functions which are +exposed only for specific use cases like debuggers and profilers. + + +With the extern keyword +----------------------- + +Functions in ``Include/internal/`` defined with the ``extern`` keyword +*must not and can not* be used outside the CPython code base. Only +built-in stdlib extensions (built with the ``Py_BUILD_CORE_BUILTIN`` +macro defined) can use such functions. + +When in doubt, new internal C functions should be defined in +``Include/internal`` using the ``extern`` keyword. + +Private names +-------------- + +Any API named with a leading underscore is also considered internal. +There are two main use cases for using such names rather than putting the +definition in ``Include/internal/`` (or directly in a ``.c`` file): + +* Internal helpers for other public API; users should not use these directly; +* “Provisional” API, included in a Python release to test real-world usage + of new API. Such names should be renamed when stabilized; preferably with + a macro aliasing the old name to the new one. + See `"Finalizing the API" in PEP 590`_ for an example. + +.. _"Finalizing the API" in PEP 590: https://www.python.org/dev/peps/pep-0590/#finalizing-the-api + + +.. _public-capi: + +Public C API +============ + +CPython's public C API is available when ``Python.h`` is included normally +(that is, without defining macros to select the other variants). + +It should be defined in ``Include/cpython/`` (unless part of the Limited API, +see below). + +Guidelines for expanding/changing the public API: + +- Make sure the new API follows reference counting conventions. + (Following them makes the API easier to reason about, and easier use + in other Python implementations.) + + - Functions *must not* steal references + - Functions *must not* return borrowed references + - Functions returning references *must* return a strong reference + +- Make sure the ownership rules and lifetimes of all applicable struct + fields, arguments and return values are well defined. + + +Limited API +=========== + +The Limited API is a subset of the C API designed to guarantee ABI +stability across Python 3 versions. +Defining the macro ``Py_LIMITED_API`` will limit the exposed API to +this subset. + +No changes that break the Stable ABI are allowed. + +The Limited API should be defined in ``Include/``, excluding the +``cpython`` and ``internal`` subdirectories. + +Guidelines for changing the Limited API +--------------------------------------- + +- Guidelines for the general :ref:`public-capi` apply. + +- New Limited API should only be defined if ``Py_LIMITED_API`` is set + to the version the API was added in or higher. + (See below for the proper ``#if`` guard.) + +- All parameter types, return values, struct members, etc. need to be part + of the Limited API. + + - Functions that deal with ``FILE*`` (or other types with ABI portability + issues) should not be added. + +- Think twice when defining macros. + + - Macros should not expose implementation details + - Functions must be exported as actual functions, not (only) + as functions-like macros. + - If possible, avoid macros. This makes the Limited API more usable in + languages that don't use the C preprocessor. + +- Please start a public discussion before expanding the Limited API + +- The Limited API and must follow standard C, not just features of currently + supported platforms. The exact C dialect is described in :pep:`7`. + + - Documentation examples (and more generally: the intended use of the API) + should also follow standard C. + - In particular, do not cast a function pointer to ``void*`` (a data pointer) + or vice versa. + +- Think about ease of use for the user. + + - In C, ease of use itself is not very important; what is useful is + reducing boilerplate code needed to use the API. Bugs like to hide in + boiler plates. + + - If a function will be often called with specific value for an argument, + consider making it default (used when ``NULL`` is passed in). + - The Limited API needs to be well documented. + +- Think about future extensions + + - If it's possible that future Python versions will need to add a new + field to your struct, make sure it can be done. + - Make as few assumptions as possible about implementation details that + might change in future CPython versions or differ across C API + implementations. The most important CPython-specific implementation + details involve: + + - The GIL + - :ref:`Garbage collection ` + - Memory layout of PyObject, lists/tuples and other structures + +If following these guidelines would hurt performance, add a fast function +(or macro) to the non-limited API and a stable equivalent to the Limited +API. + +If anything is unclear, or you have a good reason to break the guidelines, +consider discussing the change at the `capi-sig`_ mailing list. + +.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/ + +Adding a new definition to the Limited API +------------------------------------------ + +- Add the declaration to a header file directly under ``Include/``, into a + block guarded with the following: + + .. code-block:: c + + #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000 + + with the ``yy`` corresponding to the target CPython version, e.g. + ``0x030A0000`` for Python 3.10. +- Append an entry to the Stable ABI manifest, ``Misc/stable_abi.txt``. +- Regenerate the autogenerated files using ``make regen-limited-abi``. + On platforms without ``make``, run this command directly: + + .. code-block:: shell + + ./python ./Tools/scripts/stable_abi.py --generate-all ./Misc/stable_abi.txt + +- Build Python and check the using ``make check-limited-abi``. + On platforms without ``make``, run this command directly: + + .. code-block:: shell + + ./python ./Tools/scripts/stable_abi.py --all ./Misc/stable_abi.txt diff --git a/index.rst b/index.rst index 31aa398d4..126a3428f 100644 --- a/index.rst +++ b/index.rst @@ -324,6 +324,7 @@ Full Table of Contents compiler garbage_collector extensions + c-api coverity clang buildworker From 2cbf00f29b4d740f9d75544c2043adea5798e593 Mon Sep 17 00:00:00 2001 From: "T. Wouters" Date: Mon, 10 May 2021 20:35:28 +0200 Subject: [PATCH 025/793] Expand the section on renaming to the 'main' branch. (#687) Expand the section on renaming to the 'main' branch by including instructions to rename local forks on GitHub. --- gitbootcamp.rst | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/gitbootcamp.rst b/gitbootcamp.rst index 9cb76ac2f..d77a52762 100644 --- a/gitbootcamp.rst +++ b/gitbootcamp.rst @@ -157,9 +157,30 @@ You may specify more than one branch for deletion. Renaming Branch --------------- -The CPython repository's default branch was renamed from ``master`` to ``main`` -after the Python 3.10b1 release. If you had cloned the repository before this -change, you can rename your local branch as follows:: +The CPython repository's default branch was renamed from ``master`` to +``main`` after the Python 3.10b1 release. + +If you have a fork on GitHub (as described in :ref:`fork-cpython`) that was +created before the rename, you should visit the GitHub page for your fork to +rename the branch there. You only have to do this once. GitHub should +provide you with a dialog for this. If it doesn't (or the dialog was already +dismissed), you can rename the branch in your fork manually `by following +these GitHub instructions `__ + +After renaming the branch in your fork, you need to update any local clones +as well. This only has to be done once per clone:: + + git branch -m master main + git fetch origin + git branch -u origin/main main + git remote set-head origin -a + +(GitHub also provides these instructions after you rename the branch.) + +If you do not have a fork on GitHub, but rather a direct clone of the main +repo created before the branch rename, you still have to update your local +clones. This still only has to be done once per clone. In that case, you can +rename your local branch as follows:: git branch -m master main git fetch upstream From 75297e1b60c569eb4fee8c7f9b97af3ac05b949c Mon Sep 17 00:00:00 2001 From: Mariatta Wijaya Date: Mon, 10 May 2021 17:20:44 -0700 Subject: [PATCH 026/793] Use recent versions of Sphinx and python-docs-theme (#692) --- requirements.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/requirements.txt b/requirements.txt index 2a84c8173..98de25159 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,2 @@ -Sphinx -python-docs-theme +Sphinx==3.5.4 +python-docs-theme==2021.5 From d76a41e82327c937f03bf4fa83e376540cfbc998 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 10 May 2021 22:12:10 -0700 Subject: [PATCH 027/793] Add Irit Katriel to the developer log --- developers.csv | 41 +++++++++++++++++++++-------------------- 1 file changed, 21 insertions(+), 20 deletions(-) diff --git a/developers.csv b/developers.csv index ffee4c300..b37e5a15e 100644 --- a/developers.csv +++ b/developers.csv @@ -1,3 +1,4 @@ +Irit Katriel,iritkatriel,2021-05-10,, Batuhan Taskaya,isidentical,2020-11-08,, Brandt Bucher,brandtbucher,2020-09-14,, Lysandros Nikolaou,lysnikolaou,2020-06-29,, @@ -24,7 +25,7 @@ Xiang Zhang,zhangyangyu,2016-11-21,, Inada Naoki,methane,2016-09-26,, Xavier de Gaye,xdegaye,2016-06-03,2018-01-25,Privileges relinquished on 2018-01-25 Davin Potts,applio,2016-03-06,, -Martin Panter,vadmium,2015-08-10,, +Martin Panter,vadmium,2015-08-10,2020-11-26, Paul Moore,pfmoore,2015-03-15,, Robert Collins,rbtcollins,2014-10-16,,To work on unittest Berker Peksağ,berkerpeksag,2014-06-26,, @@ -42,8 +43,8 @@ Peter Moody,,2012-05-20,2017-02-10,For ipaddress module; did not make GitHub tra Hynek Schlawack,hynek,2012-05-14,, Richard Oudkerk,,2012-04-29,2017-02-10,For multiprocessing module; did not make GitHub transition Andrew Svetlov,asvetlov,2012-03-13,,At PyCon sprint -Petri Lehtinen,akheron,2011-10-22,, -Meador Inge,meadori,2011-09-19,, +Petri Lehtinen,akheron,2011-10-22,2020-11-12, +Meador Inge,meadori,2011-09-19,2020-11-26, Jeremy Kloth,jkloth,2011-09-12,, Sandro Tosi,sandrotosi,2011-08-01,, Alex Gaynor,alex,2011-07-18,,For PyPy compatibility (since expanded scope) @@ -51,9 +52,9 @@ Charles-François Natali,,2011-05-19,2017-02-10,Did not make GitHub transition Nadeem Vawda,,2011-04-10,2017-02-10,Did not make GitHub transition Jason R. Coombs,jaraco,2011-03-14,,For sprinting on distutils2 Ross Lagerwall,,2011-03-13,2017-02-10,Did not make GitHub transition -Eli Bendersky,eliben,2011-01-11,, +Eli Bendersky,eliben,2011-01-11,2020-11-26, Ned Deily,ned-deily,2011-01-09,, -David Malcolm,davidmalcolm,2010-10-27,, +David Malcolm,davidmalcolm,2010-10-27,2020-11-12,relinquished privileges on 2020-11-12 Tal Einat,taleinat,2010-10-04,,For IDLE Łukasz Langa,ambv,2010-09-08,, Daniel Stutzbach,,2010-08-22,2017-02-10,Did not make GitHub transition @@ -69,16 +70,16 @@ Dino Viehland,DinoV,2010-02-23,,For IronPython compatibility Larry Hastings,larryhastings,2010-02-22,, Victor Stinner,vstinner,2010-01-30,, Stefan Krah,skrah,2010-01-05,2020-10-07,For the decimal module -Doug Hellmann,dhellmann,2009-09-20,,For documentation +Doug Hellmann,dhellmann,2009-09-20,2020-11-11,For documentation; relinquished privileges on 2020-11-11 Frank Wierzbicki,,2009-08-02,2017-02-10,For Jython compatibility; did not make GitHub transition Ezio Melotti,ezio-melotti,2009-06-07,,For documentation -Philip Jenvey,pjenvey,2009-05-07,,For Jython compatibility +Philip Jenvey,pjenvey,2009-05-07,2020-11-26,For Jython compatibility Michael Foord,voidspace,2009-04-01,,For IronPython compatibility R\. David Murray,bitdancer,2009-03-30,, Chris Withers,cjw296,2009-03-08,, Tarek Ziadé,,2008-12-21,2017-02-10,For distutils module; did not make GitHub transition Hirokazu Yamamoto,,2008-08-12,2017-02-10,For Windows build; did not make GitHub transition -Armin Ronacher,mitsuhiko,2008-07-23,,For documentation toolset and ast module +Armin Ronacher,mitsuhiko,2008-07-23,2020-11-26,For documentation toolset and ast module Antoine Pitrou,pitrou,2008-07-16,, Senthil Kumaran,orsenthil,2008-06-16,,For GSoC Jesse Noller,,2008-06-16,2017-02-10,For multiprocessing module; did not make GitHub transition @@ -86,22 +87,22 @@ Jesús Cea,jcea,2008-05-13,,For bsddb module Guilherme Polo,,2008-04-24,2017-02-10,Did not make GitHub transition Jeroen Ruigrok van der Werven,,2008-04-12,2017-02-10,For documentation; did not make GitHub transition Benjamin Peterson,benjaminp,2008-03-25,,For bug triage -David Wolever,wolever,2008-03-17,,For 2to3 module -Trent Nelson,tpn,2008-03-17,, +David Wolever,wolever,2008-03-17,2020-11-21,For 2to3 module +Trent Nelson,tpn,2008-03-17,2020-11-26, Mark Dickinson,mdickinson,2008-01-06,,For maths-related work -Amaury Forgeot d'Arc,amauryfa,2007-11-09,, +Amaury Forgeot d'Arc,amauryfa,2007-11-09,2020-11-26, Christian Heimes,tiran,2007-10-31,, Bill Janssen,,2007-08-28,2017-02-10,For ssl module; did not make GitHub transition Jeffrey Yasskin,,2007-08-09,2017-02-10,Did not make GitHub transition Mark Summerfield,,2007-08-01,2017-02-10,For documentation; did not make GitHub transition -Alexandre Vassalotti,avassalotti,2007-05-21,,For GSoC +Alexandre Vassalotti,avassalotti,2007-05-21,2020-11-12,For GSoC Travis E. Oliphant,,2007-04-17,2017-02-10,Did not make GitHub transition Eric V. Smith,ericvsmith,2007-02-28,,For PEP 3101 in a sandbox Josiah Carlson,,2007-01-06,2017-02-10,For asyncore and asynchat modules; did not make GitHub transition Collin Winter,,2007-01-05,2017-02-10,For PEP access; did not make GitHub transition Richard Jones,,2006-05-23,2017-02-10,For Need for Speed sprint; did not make GitHub transition Kristján Valur Jónsson,,2006-05-17,2017-02-10,For Need for Speed sprint; did not make GitHub transition -Jack Diederich,jackdied,2006-05-17,,For Need for Speed sprint +Jack Diederich,jackdied,2006-05-17,2020-11-26,For Need for Speed sprint Steven Bethard,,2006-04-27,2017-02-10,For PEP access and SourceForge maintenance; did not make GitHub transition Gerhard Häring,,2006-04-23,2017-02-10,Did not make the GitHub transition George Yoshida,,2006-04-17,2017-02-10,For tracker administration; did not make GitHub transition @@ -115,10 +116,10 @@ Facundo Batista,facundobatista,2004-10-16,, Sean Reifschneider,,2004-09-17,2017-02-10,Did not make GitHub transition Johannes Gijsbers,,2004-08-14,2005-07-27,Privileges relinquished on 2005-07-27 Matthias Klose,doko42,2004-08-04,, -PJ Eby,pjeby,2004-03-24,, +PJ Eby,pjeby,2004-03-24,2020-11-26, Vinay Sajip,vsajip,2004-02-20,, Hye-Shik Chang,,2003-12-10,2017-02-10,Did not make GitHub transition -Armin Rigo,,2003-10-24,2012,Privileges relinquished in 2012 +Armin Rigo,,2003-10-24,2012-06-01,Privileges relinquished in 2012 Andrew McNamara,,2003-06-09,2017-02-10,Did not make GitHub transition Samuele Pedroni,,2003-05-16,2017-02-10,Did not make GitHub transition Alex Martelli,aleaxit,2003-04-22,, @@ -145,10 +146,10 @@ Steven M. Gava,,2001-06-25,2017-02-10,Did not make GitHub transition Steve Purcell,,2001-03-22,2017-02-10,Did not make GitHub transition Jim Fulton,,2000-10-06,2017-02-10,Did not make GitHub transition Ka-Ping Yee,,2000-10-03,2017-02-10,Did not make GitHub transition -Lars Gustäbel,gustaebel,2000-09-21,,For tarfile module +Lars Gustäbel,gustaebel,2000-09-21,2020-11-26,For tarfile module Neil Schemenauer,nascheme,2000-09-15,, Martin v. Löwis,,2000-09-08,2017-02-10,Did not make GitHub transition -Thomas Heller,theller,2000-09-07,, +Thomas Heller,theller,2000-09-07,2020-11-18, Moshe Zadka,,2000-07-29,2005-04-08,Privileges relinquished on 2005-04-08 Thomas Wouters,Yhg1s,2000-07-14,, Peter Schneider-Kamp,,2000-07-10,2017-02-10,Did not make GitHub transition @@ -163,12 +164,12 @@ Eric S. Raymond,,2000-06-02,2017-02-10,Did not make GitHub transition Greg Stein,,1999-11-07,2017-02-10,Did not make GitHub transition Just van Rossum,,1999-01-22,2017-02-10,Did not make GitHub transition Greg Ward,,1998-12-18,2017-02-10,Did not make GitHub transition -Andrew Kuchling,akuchling,1998-04-09,2021-02-22,Privileges relinquished on 2021-02-22 +Andrew Kuchling,akuchling,1998-04-09,, Ken Manheimer,,1998-03-03,2005-04-08,Privileges relinquished on 2005-04-08 -Jeremy Hylton,jeremyhylton,1997-08-13,, +Jeremy Hylton,jeremyhylton,1997-08-13,2020-11-26, Roger E. Masse,,1996-12-09,2017-02-10,Did not make GitHub transition Fred Drake,freddrake,1996-07-23,, Barry Warsaw,warsaw,1994-07-25,, Jack Jansen,jackjansen,1992-08-13,, -Sjoerd Mullender,sjoerdmullender,1992-08-04,, +Sjoerd Mullender,sjoerdmullender,1992-08-04,2020-11-14, Guido van Rossum,gvanrossum,1989-12-25,, From 4a87aa0b9ec33d2f070e62df420fd4aa207ff902 Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Mon, 10 May 2021 22:13:43 -0700 Subject: [PATCH 028/793] Add a link to the CoC --- coredev.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/coredev.rst b/coredev.rst index af8b8579b..195b5f559 100644 --- a/coredev.rst +++ b/coredev.rst @@ -81,7 +81,7 @@ The steps to gaining commit privileges are: https://github.com/python/voters/ - Your preferred email address to subscribe to python-committers with - - A reminder about the Code of Conduct and to report issues to the PSF + - A reminder about the `Code of Conduct`_ and to report issues to the PSF Conduct WG 6. Once you have provided the pertinent details, your various new privileges @@ -91,6 +91,8 @@ The steps to gaining commit privileges are: :ref:`developers` 9. An announcement email by the steering council member handling your new membership will be sent to python-committers + +.. _Code of Conduct: https://www.python.org/psf/conduct/ Mailing Lists From 05426d053643fd8b057e7ab272a69ea3fd99e4fa Mon Sep 17 00:00:00 2001 From: Irit Katriel Date: Tue, 11 May 2021 14:03:33 +0100 Subject: [PATCH 029/793] Added myself as traceback expert (#694) --- experts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experts.rst b/experts.rst index 1f0c96192..11d65b982 100644 --- a/experts.rst +++ b/experts.rst @@ -236,7 +236,7 @@ tkinter gpolo, serhiy.storchaka token tokenize meador.inge trace belopolsky -traceback +traceback iritkatriel tracemalloc vstinner tty twouters* turtle gregorlingl, willingc From ad91664f51ddda8fc3ea095c0a3c9d92dd2d9b73 Mon Sep 17 00:00:00 2001 From: "Kerwin(Kaihui) Sun" Date: Tue, 11 May 2021 23:11:34 +0800 Subject: [PATCH 030/793] Remove py27 in setup step (#691) --- index.rst | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/index.rst b/index.rst index 126a3428f..733938dbe 100644 --- a/index.rst +++ b/index.rst @@ -47,8 +47,7 @@ instructions please see the :ref:`setup guide `. ./python -m test -j3 On :ref:`most ` Mac OS X systems, replace :file:`./python` - with :file:`./python.exe`. On Windows, use :file:`python.bat`. With Python - 2.7, replace ``test`` with ``test.regrtest``. + with :file:`./python.exe`. On Windows, use :file:`python.bat`. 5. Create a new branch where your work for the issue will go, e.g.:: From d292422120918888302c6116932f873475d503c4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?St=C3=A9phane=20Wirtel?= Date: Wed, 12 May 2021 10:51:33 +0200 Subject: [PATCH 031/793] Pip updates during the installation of the virtual env (#688) --- Makefile | 1 + 1 file changed, 1 insertion(+) diff --git a/Makefile b/Makefile index b9f2a0d9c..3f3963296 100644 --- a/Makefile +++ b/Makefile @@ -43,6 +43,7 @@ clean: venv: $(PYTHON) -m venv venv + ./venv/bin/python3 -m pip install --upgrade pip ./venv/bin/python3 -m pip install -r requirements.txt html: venv From 3e2805a576bd466767137234a21c8d0a26f9b292 Mon Sep 17 00:00:00 2001 From: Tal Einat <532281+taleinat@users.noreply.github.com> Date: Sat, 15 May 2021 22:41:04 +0300 Subject: [PATCH 032/793] remove taleinat as an expert for the statistics module (#697) --- experts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experts.rst b/experts.rst index 11d65b982..001ff642a 100644 --- a/experts.rst +++ b/experts.rst @@ -211,7 +211,7 @@ spwd sqlite3 ghaering ssl janssen, christian.heimes, dstufft, alex stat christian.heimes -statistics steven.daprano, rhettinger, taleinat +statistics steven.daprano, rhettinger string stringprep struct mark.dickinson, meador.inge From 51591ad2849eef0204149530a7b562ea897c40c4 Mon Sep 17 00:00:00 2001 From: Bonifacio de Oliveira Date: Mon, 17 May 2021 05:48:11 -0300 Subject: [PATCH 033/793] minor grammar improvement (GH-696) --- docquality.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docquality.rst b/docquality.rst index 2f26f7c55..8e20d0c7b 100644 --- a/docquality.rst +++ b/docquality.rst @@ -99,7 +99,7 @@ lives in a `separate repository`_ and bug reports should be submitted to the Our devguide workflow uses continuous integration and deployment so changes to the devguide are normally published when the pull request is merged. Changes -to CPython documentation follows the workflow of a CPython release and is +to CPython documentation follow the workflow of a CPython release and are published in the release. From 6e960efe0c28d36c7810fdf64d9d3300c22a0397 Mon Sep 17 00:00:00 2001 From: "Kerwin(Kaihui) Sun" Date: Mon, 17 May 2021 16:53:26 +0800 Subject: [PATCH 034/793] Remove py27 notes in Setup Page. (#699) --- setup.rst | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-) diff --git a/setup.rst b/setup.rst index 385f142f2..1a6dde4df 100644 --- a/setup.rst +++ b/setup.rst @@ -266,12 +266,6 @@ Visual Studio to continue development. See the `readme`_ for more details on what other software is necessary and how to build. -.. note:: **Python 2.7** uses Microsoft Visual Studio 2008, which is most easily - obtained through an MSDN subscription. To use the build files in the - `PCbuild directory`_ you will also need Visual Studio 2010, see the `2.7 - readme`_ for more details. If you have VS 2008 but not 2010 you can use the - build files in the `PC/VS9.0 directory`_, see the `VS9 readme`_ for details. - .. note:: If you are using the Windows Subsystem for Linux (WSL), clone the repository from a native Windows terminal program like cmd.exe command prompt or PowerShell as well as use a build of git targeted for Windows, e.g., the @@ -281,11 +275,6 @@ to build. .. _this documentation: https://cpython-core-tutorial.readthedocs.io/en/latest/build_cpython_windows.html .. _Visual Studio 2017: https://www.visualstudio.com/ .. _readme: https://github.com/python/cpython/blob/main/PCbuild/readme.txt -.. _PCbuild directory: https://github.com/python/cpython/tree/2.7/PCbuild/ -.. _2.7 readme: https://github.com/python/cpython/blob/2.7/PCbuild/readme.txt -.. _PC/VS9.0 directory: https://github.com/python/cpython/tree/2.7/PC/VS9.0/ -.. _VS9 readme: https://github.com/python/cpython/blob/2.7/PC/VS9.0/readme.txt - .. _build-dependencies: @@ -415,9 +404,7 @@ and ``make``:: There will sometimes be optional modules added for a new release which won't yet be identified in the OS level build dependencies. In those cases, -just ask for assistance on the core-mentorship list. If working on bug -fixes for Python 2.7, use ``python`` in place of ``python3`` in the above -commands. +just ask for assistance on the core-mentorship list. Explaining how to build optional dependencies on a UNIX based system without root access is beyond the scope of this guide. From c8f80aea8217604759360c7f13a8a484042500f5 Mon Sep 17 00:00:00 2001 From: Bonifacio de Oliveira Date: Tue, 18 May 2021 19:26:00 -0300 Subject: [PATCH 035/793] minor grammar improvement (#703) --- triaging.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/triaging.rst b/triaging.rst index 60025394b..4724c9f9c 100644 --- a/triaging.rst +++ b/triaging.rst @@ -57,7 +57,7 @@ Becoming a member of the Python triage team Any Python core developers are welcome to invite a Python contributor to the Python triage team. Do note that the responsibilities of a Python triager -is more elevated than a developer on bpo. For example, the Python triager +are more elevated than a developer on bpo. For example, the Python triager has access to more repositories than just CPython. Triagers will be responsible to handle not just issues, but also pull requests, and even managing backports. From d84dd7a9b0b206a139473a4f8b0ffedc4e9d8328 Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Tue, 25 May 2021 06:29:33 +0800 Subject: [PATCH 036/793] Bring compiler docs up to speed with Python 3.10 (#706) --- compiler.rst | 218 ++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 165 insertions(+), 53 deletions(-) diff --git a/compiler.rst b/compiler.rst index 40384cb07..9cc2a1644 100644 --- a/compiler.rst +++ b/compiler.rst @@ -139,8 +139,8 @@ that is needed to completely free all memory used by the compiler. In general, unless you are working on the critical core of the compiler, memory management can be completely ignored. But if you are working at either the very beginning of the compiler or the end, you need to care about how the arena -works. All code relating to the arena is in either :file:`Include/pyarena.h` or -:file:`Python/pyarena.c`. +works. All code relating to the arena is in either +:file:`Include/Internal/pycore_pyarena.h` or :file:`Python/pyarena.c`. ``PyArena_New()`` will create a new arena. The returned ``PyArena`` structure will store pointers to all memory given to it. This does the bookkeeping of @@ -159,49 +159,131 @@ are very rare. However, if you've allocated a PyObject, you must tell the arena about it by calling ``PyArena_AddPyObject()``. -Parse Tree to AST ------------------ +Source Code to AST +------------------ + +The AST is generated from source code using the function +``_PyParser_ASTFromString()`` or ``_PyParser_ASTFromFile()`` +(from :file:`Parser/peg_api.c`) depending on the input type. + +After some checks, a helper function in :file:`Parser/parser.c` begins applying +production rules on the source code it receives; converting source code to +tokens and matching these tokens recursively to their corresponding rule. The +rule's corresponding rule function is called on every match. These rule +functions follow the format :samp:`xx_rule`. Where *xx* is the grammar rule +that the function handles and is automatically derived from +:file:`Grammar/python.gram` by :file:`Tools/peg_generator/pegen/c_generator.py`. + +Each rule function in turn creates an AST node as it goes along. It does this +by allocating all the new nodes it needs, calling the proper AST node creation +functions for any required supporting functions and connecting them as needed. +This continues until all nonterminal symbols are replaced with terminals. If an +error occurs, the rule functions backtrack and try another rule function. If +there are no more rules, an error is set and the parsing ends. + +The AST node creation helper functions have the name :samp:`_PyAST_{xx}` +where *xx* is the AST node that the function creates. These are defined by the +ASDL grammar and contained in :file:`Python/Python-ast.c` (which is generated by +:file:`Parser/asdl_c.py` from :file:`Parser/Python.asdl`). This all leads to a +sequence of AST nodes stored in ``asdl_seq`` structs. + +To demonstrate everything explained so far, here's the +rule function responsible for a simple named import statement such as +``import sys``. Note that error-checking and debugging code has been +omitted. Removed parts are represented by ``...``. +Furthermore, some comments have been added for explanation. These comments +may not be present in the actual code. + +.. code-block:: c -The AST is generated from the parse tree (see :file:`Python/ast.c`) using the -function ``PyAST_FromNode()``. + // This is the production rule (from python.gram) the rule function + // corresponds to: + // import_name: 'import' dotted_as_names + static stmt_ty + import_name_rule(Parser *p) + { + ... + stmt_ty _res = NULL; + { // 'import' dotted_as_names + ... + Token * _keyword; + asdl_alias_seq* a; + // The tokenizing steps. + if ( + (_keyword = _PyPegen_expect_token(p, 513)) // token='import' + && + (a = dotted_as_names_rule(p)) // dotted_as_names + ) + { + ... + // Generate an AST for the import statement. + _res = _PyAST_Import ( a , ...); + ... + goto done; + } + ... + } + _res = NULL; + done: + ... + return _res; + } -The function begins a tree walk of the parse tree, creating various AST -nodes as it goes along. It does this by allocating all new nodes it -needs, calling the proper AST node creation functions for any required -supporting functions, and connecting them as needed. -Do realize that there is no automated nor symbolic connection between -the grammar specification and the nodes in the parse tree. No help is -directly provided by the parse tree as in yacc. +To improve backtracking performance, some rules (chosen by applying a +``(memo)`` flag in the grammar file) are memoized. Each rule function checks if +a memoized version exists and returns that if so, else it continues in the +manner stated in the previous paragraphs. -For instance, one must keep track of which node in the parse tree -one is working with (e.g., if you are working with an 'if' statement -you need to watch out for the ':' token to find the end of the conditional). +There are macros for creating and using ``asdl_xx_seq *`` types, where *xx* is +a type of the ASDL sequence. Three main types are defined +manually -- ``generic``, ``identifier`` and ``int``. These types are found in +:file:`Python/asdl.c` and its corresponding header file +:file:`Include/Internal/pycore_asdl.h`. Functions and macros +for creating ``asdl_xx_seq *`` types are as follows: -The functions called to generate AST nodes from the parse tree all have -the name :samp:`ast_for_{xx}` where *xx* is the grammar rule that the function -handles (``alias_for_import_name`` is the exception to this). These in turn -call the constructor functions as defined by the ASDL grammar and -contained in :file:`Python/Python-ast.c` (which was generated by -:file:`Parser/asdl_c.py`) to create the nodes of the AST. This all leads to a -sequence of AST nodes stored in ``asdl_seq`` structs. +``_Py_asdl_generic_seq_new(Py_ssize_t, PyArena *)`` + Allocate memory for an ``asdl_int_seq`` of the specified length +``_Py_asdl_identifier_seq_new(Py_ssize_t, PyArena *)`` + Allocate memory for an ``asdl_identifier_seq`` of the specified length +``_Py_asdl_int_seq_new(Py_ssize_t, PyArena *)`` + Allocate memory for an ``asdl_generic_seq`` of the specified length -Function and macros for creating and using ``asdl_seq *`` types as found -in :file:`Python/asdl.c` and :file:`Include/asdl.h` are as follows: +In addition to the three types mentioned above, some ASDL sequence types are +automatically generated by :file:`Parser/asdl_c.py` and found in +:file:`Include/Internal/pycore_ast.h`. Macros for using both manually defined +and automatically generated ASDL sequence types are as follows: -``_Py_asdl_seq_new(Py_ssize_t, PyArena *)`` - Allocate memory for an ``asdl_seq`` for the specified length -``asdl_seq_GET(asdl_seq *, int)`` +``asdl_seq_GET(asdl_xx_seq *, int)`` + Get item held at a specific position in an ``asdl_xx_seq`` +``asdl_seq_SET(asdl_xx_seq *, int, stmt_ty)`` + Set a specific index in an ``asdl_xx_seq`` to the specified value + +Untyped counterparts exist for some of the typed macros. These are useful +when a function needs to manipulate a generic ASDL sequence: + +``asdl_seq_GET_UNTYPED(asdl_seq *, int)`` Get item held at a specific position in an ``asdl_seq`` -``asdl_seq_SET(asdl_seq *, int, stmt_ty)`` +``asdl_seq_SET_UNTYPED(asdl_seq *, int, stmt_ty)`` Set a specific index in an ``asdl_seq`` to the specified value ``asdl_seq_LEN(asdl_seq *)`` - Return the length of an ``asdl_seq`` + Return the length of an ``asdl_seq`` or ``asdl_xx_seq`` + +Note that typed macros and functions are recommended over their untyped +counterparts. Typed macros carry out checks in debug mode and aid +debugging errors caused by incorrectly casting from ``void *``. If you are working with statements, you must also worry about keeping track of what line number generated the statement. Currently the line number is passed as the last parameter to each ``stmt_ty`` function. +.. versionchanged:: 3.9 + The new PEG parser generates an AST directly without creating a + parse tree. ``Python/ast.c`` is now only used to validate the AST for + debugging purposes. + +.. seealso:: :pep:`617` (PEP 617 -- New PEG parser for CPython) + Control Flow Graphs ------------------- @@ -248,13 +330,13 @@ global). With that done, the second pass essentially flattens the CFG into a list and calculates jump offsets for final output of bytecode. The conversion process is initiated by a call to the function -``PyAST_CompileObject()`` in :file:`Python/compile.c`. This function does both the +``_PyAST_Compile()`` in :file:`Python/compile.c`. This function does both the conversion of the AST to a CFG and outputting final bytecode from the CFG. The AST to CFG step is handled mostly by two functions called by -``PyAST_CompileObject()``; ``PySymtable_BuildObject()`` and ``compiler_mod()``. The former +``_PyAST_Compile()``; ``_PySymtable_Build()`` and ``compiler_mod()``. The former is in :file:`Python/symtable.c` while the latter is in :file:`Python/compile.c`. -``PySymtable_BuildObject()`` begins by entering the starting code block for the +``_PySymtable_Build()`` begins by entering the starting code block for the AST (passed-in) and then calling the proper :samp:`symtable_visit_{xx}` function (with *xx* being the AST node type). Next, the AST tree is walked with the various code blocks that delineate the reach of a local variable @@ -369,6 +451,9 @@ command before adding the new bytecode target to :file:`Python/ceval.c` will result in an error. You should only run ``make regen-importlib`` after the new bytecode target has been added. +.. note:: On Windows, running the ``./build.bat`` script will automatically + regenerate the required files without requiring additional arguments. + Finally, you need to introduce the use of the new bytecode. Altering :file:`Python/compile.c` and :file:`Python/ceval.c` will be the primary places to change. You must add the case for a new opcode into the 'switch' @@ -419,7 +504,28 @@ Important Files asdl_c.py "Generate C code from an ASDL description." Generates - :file:`Python/Python-ast.c` and :file:`Include/Python-ast.h`. + :file:`Python/Python-ast.c` and :file:`Include/Internal/pycore_ast.h`. + + parser.c + The new PEG parser introduced in Python 3.9. + Generated by :file:`Tools/peg_generator/pegen/c_generator.py` + from the grammar :file:`Grammar/python.gram`. Creates the AST from + source code. Rule functions for their corresponding production rules + are found here. + + peg_api.c + Contains high-level functions which are used by the interpreter to + create an AST from source code . + + pegen.c + Contains helper functions which are used by functions in + :file:`Parser/parser.c` to construct the AST. Also contains helper + functions which help raise better error messages when parsing source + code. + + pegen.h + Header file for the corresponding :file:`Parser/pegen.c`. Also contains + definitions of the ``Parser`` and ``Token`` structs. + Python/ @@ -437,7 +543,7 @@ Important Files identifier. Used by :file:`Python-ast.c` for marshalling AST nodes. ast.c - Converts Python's parse tree into the abstract syntax tree. + Used for validating the AST. ast_opt.c Optimizes the AST. @@ -469,31 +575,37 @@ Important Files + Include/ - Python-ast.h - Contains the actual definitions of the C structs as generated by - :file:`Python/Python-ast.c`. - "Automatically generated by :file:`Parser/asdl_c.py`". - - asdl.h - Header for the corresponding :file:`Python/ast.c`. - - ast.h - Declares ``PyAST_FromNode()`` external (from :file:`Python/ast.c`). - code.h Header file for :file:`Objects/codeobject.c`; contains definition of ``PyCodeObject``. - symtable.h - Header for :file:`Python/symtable.c`. ``struct symtable`` and - ``PySTEntryObject`` are defined here. - - pyarena.h - Header file for the corresponding :file:`Python/pyarena.c`. - opcode.h One of the files that must be modified if :file:`Lib/opcode.py` is. + + Internal/ + + pycore_ast.h + Contains the actual definitions of the C structs as generated by + :file:`Python/Python-ast.c`. + "Automatically generated by :file:`Parser/asdl_c.py`". + + pycore_asdl.h + Header for the corresponding :file:`Python/ast.c` + + pycore_ast.h + Declares ``_PyAST_Validate()`` external (from :file:`Python/ast.c`). + + pycore_symtable.h + Header for :file:`Python/symtable.c`. ``struct symtable`` and + ``PySTEntryObject`` are defined here. + + pycore_parser.h + Header for the corresponding :file:`Parser/peg_api.c`. + + pycore_pyarena.h + Header file for the corresponding :file:`Python/pyarena.c`. + + + Objects/ codeobject.c From 34b2f7e64c8b45686af54ed8aa93ba63d1fdf601 Mon Sep 17 00:00:00 2001 From: Victor Stinner Date: Wed, 26 May 2021 13:42:10 +0200 Subject: [PATCH 037/793] Migrate from Freenode to LiberaChat (#708) --- buildbots.rst | 4 ++-- communication.rst | 11 +++++------ documenting.rst | 3 ++- help.rst | 4 ++-- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/buildbots.rst b/buildbots.rst index 58535ca1e..ca342938d 100644 --- a/buildbots.rst +++ b/buildbots.rst @@ -52,8 +52,8 @@ There are three ways of visualizing recent build results: you. You can also access builder information by clicking on the builder status bubbles in the top line. -If you like IRC, having an IRC client open to the #python-dev channel on -irc.freenode.net is useful. Any time a builder changes state (last build +If you like IRC, having an IRC client open to the #python-dev-notifs channel on +irc.libera.chat is useful. Any time a builder changes state (last build passed and this one didn't, or vice versa), a message is posted to the channel. Keeping an eye on the channel after pushing a changeset is a simple way to get notified that there is something you should look in to. diff --git a/communication.rst b/communication.rst index 90b5a1027..7d39972ef 100644 --- a/communication.rst +++ b/communication.rst @@ -50,7 +50,7 @@ to any issue, subscribe to python-bugs-list_. General Python questions should go to `python-list`_ or `tutor`_ or similar resources, such as StackOverflow_ or the ``#python`` IRC channel -on Freenode_. +on Libera.Chat_. `Core-Workflow `_ mailing list is the place to discuss and work on improvements to the CPython @@ -72,7 +72,7 @@ RSS feed readers. .. _python-list: https://mail.python.org/mailman/listinfo/python-list .. _tutor: https://mail.python.org/mailman/listinfo/tutor .. _StackOverflow: https://stackoverflow.com/ -.. _Freenode: http://freenode.net/ +.. _Libera.Chat: https://libera.chat/ Zulip ----- @@ -85,10 +85,9 @@ IRC Some core developers enjoy spending time on IRC discussing various issues regarding Python's development in the ``#python-dev`` channel on -``irc.freenode.net``. This is not a place to ask for help with Python, but to -discuss issues related to Python's own development. You can use Freenode's -`Web interface `_ if you don't have an IRC -client. +``irc.libera.chat``. This is not a place to ask for help with Python, but to +discuss issues related to Python's own development. See also the +``#python-dev-notifs`` channel for bots notifications. Blogs diff --git a/documenting.rst b/documenting.rst index 5c712a001..a0521ce9e 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1736,7 +1736,8 @@ How to get help Discussions about translations occur on the `doc-sig `_ mailing list, -and there's a freenode IRC channel, ``#python-doc``. +and there's a `Libera.Chat IRC `_ channel, +``#python-doc``. Translation FAQ diff --git a/help.rst b/help.rst index b0c726cdc..dea37d0f9 100644 --- a/help.rst +++ b/help.rst @@ -41,7 +41,7 @@ Ask #python-dev --------------- If you are comfortable with IRC you can try asking on ``#python-dev`` (on -the `freenode`_ network). Typically there are a number of experienced +the `Libera.Chat`_ network). Typically there are a number of experienced developers, ranging from triagers to core developers, who can answer questions about developing for Python. As with the mailing lists, ``#python-dev`` is for questions involving the development *of* Python @@ -52,7 +52,7 @@ whereas ``#python`` is for questions concerning development *with* Python. You may not be able to access the history of this channel, so it cannot be used as a "knowledge base" of sorts. -.. _freenode: https://freenode.net/ +.. _Libera.Chat: https://libera.chat/ Zulip ----- From cc033adeb955ba0dd5fd246c459de75be2f5199b Mon Sep 17 00:00:00 2001 From: Jelle Zijlstra Date: Wed, 12 May 2021 07:24:16 -0700 Subject: [PATCH 038/793] fix typo in status for 3.10 --- index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.rst b/index.rst index 733938dbe..7c7810ca8 100644 --- a/index.rst +++ b/index.rst @@ -99,7 +99,7 @@ Status of Python branches +==================+==============+=============+================+================+=======================+ | main | *TBD* | features | *TBD* | *TBD* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ -| 3.10 | :pep:`619` | prerelase | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +| 3.10 | :pep:`619` | prerelease | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.9 | :pep:`596` | bugfix | 2020-10-05 | *TBD* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ From d6714a4c1ce1404fde158f4faa93c6fe0ab15feb Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Fri, 4 Jun 2021 18:54:24 +0100 Subject: [PATCH 039/793] Switch the theme to 'furo' (#679) --- conf.py | 18 ++++-------------- python-logo.png | Bin 0 -> 9955 bytes requirements.txt | 3 ++- 3 files changed, 6 insertions(+), 15 deletions(-) create mode 100644 python-logo.png diff --git a/conf.py b/conf.py index 407109588..468fe1f17 100644 --- a/conf.py +++ b/conf.py @@ -29,7 +29,7 @@ # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['sphinx.ext.intersphinx', 'sphinx.ext.todo'] +extensions = ['sphinx.ext.intersphinx', 'sphinx.ext.todo', 'sphinx_copybutton'] intersphinx_mapping = {'python': ('https://docs.python.org/3', None)} todo_include_todos = True @@ -93,11 +93,8 @@ # -- Options for HTML output --------------------------------------------------- # Use the upstream python-docs-theme -html_theme = 'python_docs_theme' -html_theme_options = { - 'collapsiblesidebar': True, - 'issues_url': 'https://github.com/python/devguide/issues/new', -} +html_theme = 'furo' +html_theme_options = {} # The name for this set of Sphinx documents. If None, it defaults to @@ -107,13 +104,6 @@ # Path to find HTML templates. templates_path = ['tools/templates'] -# Custom sidebar templates, filenames relative to this file. -html_sidebars = { - # Defaults taken from http://www.sphinx-doc.org/en/stable/config.html#confval-html_sidebars - # Removes the quick search block - '**': ['localtoc.html', 'globaltoc.html', 'relations.html', 'customsourcelink.html'], -} - # Additional static files. html_static_path = ['tools/static'] @@ -122,7 +112,7 @@ # The name of an image file (relative to this directory) to place at the top # of the sidebar. -#html_logo = None +html_logo = "python-logo.png" # The name of an image file (within the static path) to use as favicon of the # docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 diff --git a/python-logo.png b/python-logo.png new file mode 100644 index 0000000000000000000000000000000000000000..49ea8f5ba9252776c0a9e0ab37d21700b42e3b2b GIT binary patch literal 9955 zcmXxK2|QH)_dkB`btf}Jj2U~>SVAHc#_rnpB^05Wq-4pyrgH5m)kGpxBTKSxg%VTQ zvZsU++7K!4B4nBW_5J+*kH^g8HLr8d>zwC#p7Xjh^SU?1-qvh8QIrS(z;+9BV+R0$ zAnXm`IWS3wwGfl+~h3B0G~ zleJZa5%Y~MEQId4;r#x z8_+@uX~FsIH%7Fe`?Syp>;Z%pj@81L7RaPwL>*REG1z^27{l9FRS3LCizwB=81}z|WykK(*=m3>8+G4I zT1bJqZx$`I7~^04Y&QF&9ed1nD{GA24>s&^d-k9a`=c$!Cws)4{oWd580$}sb5(4N zlc0Q!b2YT^hwKS^b)O7%zby8s6)oTn8>=SP60c|)(KYN*3wD>5x=$v%lV*M*D+AjY z_d*?P9RWJM&V`%B&!G&`>u7Wa&%P0b=d1&9A+iM6GES1g5Pc-RjUlWV^UKAxG~Ac( zl8h>AW5Z};EQ&=?PMMG&4Po%l5B|uycQ#EcYjw5I21044*Ao<7_mBcL1OQ7RClnk?((e3^?O9EV3=|Ls?{`4Jl;W!)?El|D7{8>K<|V z9Fa}47u*poeMo`)A>?$wce`i&hl*W;pKd*t_`p1CD7>>S(Zl30|K9Ml_8sAR!m{vz z%mfpO$IfxJp2`LmOc@>#iGtKu;pB;!>G6=l=xd<`)2pQ4-$fTULw`0-H%*rm7n3|n z%NItUS~cxVZfPEx58W2u+uJLC^ytx8#M#+wmi?Uh=DUCSvbN5+`qRq$H5E_re0oEV zY(DOciPs(xm=#I){CA>a-zks31*)|Vt;<>k&V&zZH=S+nl0Sdz{)I1lpXgrgyEd>m zcC~P9SGt+ZhPbh zXn2*xt0n6=`$u(g>9`Be0g#tM{CB0hd*Sx6f!Tb_ePR2wSd3o#^B~|6lnN=Rw7xhtWKf}11ay(sX zY2MD$2)&6|vVz~=`*T7h)^}rmdqwM)uP9o$`0hgd6CeM2(FOi-Qtg_( z=LlW9s}WIA-wc^}tS(N^ySVo_u~ZL$(}#m2t@c>9$$Z+BV+o%lVIosA*$ zPXo0(b=3MeeTSG%5W$@ohe zXrShzwesN0moJm6tr=6UffMoTBxRkZv${oz-mM?*C80Y~ak=JJh-+$X_K@MnDmP%f&-JVf+7au;AkD3=e+xDoCf3oW! zIDCLYN)bbzsvPS2a~P2xydnT)MOQLIhL~IOx&IWuK10teDr3>-vK}hS`EdBP4Xk zPOdt~cL%AMFx;9`_03sR#4h>i>rX+dS{xAZ(zUgN{^bVD>#j)WFSk`u#rc+ZVNaut zDaxC#Mfu%BNK6XJN&H<+l%WSfc$faApI}&ADD+QN`%kIPS|V9lZ_~)h{kEk%xGY|9 z?Z^Amd2s43R;s-S=T#uH?^CW%i#gQZQLVw`ZX5knZM$)c>sJ{9?K>fCUy$U%J)`>G z^AAyaa&R{{$K_cOu91MIqA8L&t1#T@XtOSTUBTY4`Z0;$j8&+ib;-#Qya4)VcfW0H zYet{tKmta4+Hi{oPb+->G?>FnP;80^@QhZxl}Y#?7Xtp7N>?zC zE_A&Cqo_=*=KQ@Z=DdhEzXgCl!$-w4?~vZj3vEqESyG!l+#d%~H#zWu0b+tG&=rtc zb3Y*~0AE%#X3rW?Z40{p{#Z5UCk?GYzk9!YBmMYg9034vAP4y7&9RLuPB;?^00l!y z5O;g*;OS48X)2J2e*b>>cW#v=^)5hSkeTGbB_E+rXN`vLZa_o;B!HIS!#bbPkGDpj z${6zlkPoE#OWH)(ESrCGO`P(~~_b%;n#u0-%{gg3%i4uV1VpF&W+#8e# zVmR=FuF*1 z{%czh&Yr+huz^Me0uxFB(txDBJq9-(IYC$R~3~|gx8*b3w9*11x>%0Y!It8F37$8jx^Z_<8f!uK@!b6_i3&6qG0WZut1jbjU zpOl%Lu!NLiCX{n_?OtlR(4tv)@@0|NLBno*pb%4Xp>^Wxs^ML?djC7Fz2%OXnZJi` zUHJ9vKdGCqiI5Dg$8#U+64pG9yf_7~FU|V8_crVX5C8Q2b587Gq~)M>_3k6S`=7Nd z30%$5(^=0*LoHDi@Eq73H6KywS-GNTXOwKU^yLeU_-ANP2A-iP@1gtn_0?bQmuz{&7P-UlaW*2!Nqb zvjlNK29cVq$wOb~UDfUq##eEYN%T8_4YUoiJxCP3lbgG~Na*&8{n^Taud+u4K`xxN zt^a1>BgoQo8Ea~Hmi}8{&luW9MQ;GQkRY)^qfRsbR*fj0E`uZ^Pasn@ zxZt z(}LP7BXJUpck6vjx%HHa{x&mvBfvDl%95@HSx*rCE|6HRtd_U%em}m~0t-vSQ|hT> zBo@XB;0AI$?bQ*ui2%`H609Joth`W{@=Rip5KLkkpyg=nEw5sNh6EZ9sKxf=B!uF) zx$XP$R&sEU0a64l5wjpP8n;HL$dSpxtP<{FS@9Ya=hH*`IO;6Mt~ zg!M#5;y$|YRB4WH0Ri+uQFw$HX--|28TtaiRYFXez|nub2HU|;!;9lm04RphtSm(0 zNvB=uf=H}`s5;04y=akXapHj@iCGhc252%W9A!9@0NjP9=6xU?xWG`$(f$K3sDtZ? zDU)PbAd!`uH6%EYtlVO1L7i8xzA^cNq_la9wIsCp}5@&3j4Igk$)dnF6ba>m`K z9r5Rp$BnHwem?aQvqXZQU1srIgDTvk4mO**LtqoG*el*7Ig;2AY;v)0%tnf?(2vWV zlAlbM7l`d|APTQmmj%4Vfw{Ob-Qr{U6FrAmopuE=PhSwh+EkqTNrlOsC&51!N8UQd zdeQcinbK8~^a7KAZo9|bNr9^UxXx{{Uq7nJL;bZMcOE7Ri`?+1)ENQ+;IGmc*UjyH zkg1+i1rY(vz=-3{cDTD!3UuQd4JRUo>&zJEWxa0-^smloj>cV?m#6a3753=UUH9ZG z5AaO(zmPMJmul=y;2O*;wk|ru;Tn%1aOv-pn`rT5$v~cnyvISSItPJ*gLRh1ZeiVi zPZPa89z0WF1r{cj7wq`2&oiv|mbIhJ1DT_7Aw{D#AD53cw2c25UmK5}_>;TbtkbX0 z>ebveetbImZoU$zm^xsqmIlbiZd8nPI+0O@$?EkBfP+SRYnD)KcjnQKEL$nC6Y4Vs z*Sr_xJHDN`jdoXDI5jb?H0iueXrTjzb@W~}@QO`}kr-UU!7|B3>3}C#mTXXLBBad! z>Aa{kw)(v=w159=9QPsZ9o&g6=<*9o;E&`0OH;a)q+A!+)X&;aM)#s&eCW{z=m(tC z?nbKkyLWOq7k9q(2uraK+7w`*9Wsa0?@2bRs1%`jwD#k8bD#uxHwW~*aM}4+aA0k4 zWcm{Wx86PpU3`Fj`v)mNoW}SWMPco@IG9e1Tt9odw%&&iI@d8yoK6F*!5%Mz=?j1} z^6P!IB79CL@Rd1a+wV!%sCbCR6!QtAh7#~5SM zg6f=S^Q{DY1Dr|;%yn9<*#*X^G@$)_xVqgngttWUfbWFM85!^lk@Z>H=;1q5Eq1iok+fJlD!#3ow#M zp`64EN4{xx>Us8XW^HT-F_?cLViWqk|1|`SW9HgDndtyGs7Scnfiy|lvIf-GiadRo#Xc1Ovk>uS-1-zu&eM zy&%2hm<+Zh>dp@*qq|KmNk}xJpMpOGtXR10Ql=heO}xcxh)mA~4mdM1lb^xd0ibd^ zPyxAKTh) z;;<|2fFPGjGr^jV${aRgE$E=qWR@A?ws)oUxAhMO!B+y1_Bm`7kJ-7)XmXCPgMH6$bM9Lx7gY$6i_-mq%e9d8A=-*v1%Mw!M zAE^1_B3VpX$lV7oDuziRkRgBIW5o$rMI{cmZt)Dzj(r4ZN+TSg;;GeYdG^LCVa0?5 zNkiQhh&kl?KKzHSAZP3$%W~GY6U!fvR0xEcyDm-uN)`sBrWg}et&Ztbv(H39HbD@*K!QE zcoR@~lYtICGrhf;I{bdw(_<%1Fc-ah4)xF_G|&^2z|kvjKE$}+NjX31&VbHgY|NP> zHdVCPtkj|=3-d==hid;md&g+5t-dg<{mA5Y*Yx!{Pw!3}q+gu|ea79yH3{F$9uEAv z(%4s#9Fe<}k^l4BkWk+fuS)%fAtSd7pAXvUEtc>$%|4Ic`!me3#J56JOsq*$SO#*3 zgF~gLidEh7&Fx30bAH7r4v8OwHYru8K^WA!Z+jzQ=2aLOzZV?+Hh-sgx$xinD`+gd z&TWG9yP^-Z_!yP{tlYk6b}Y^rvTTMR{}~6Q-vQM;5w`t|)?n1cAw!(IN&?zU0Ybtg zm@Y#vkN<0Yy4*;J+|P$wT-4#HI>rmxK8zGPabR)p zWHt}Xjgy^x&5wG1f=;g-@%IgA7hL$%%C{i5lWJLq%1=Cv7ky%CBp7%dmLpXC*k;bJ zFnOk|eFH^BCt}wK;p1fDw_=%*$a+n3G5=7+^l-GGbB4)I*III(dm? z`@$;9oX(m$A|Y`VjwE%~USH+Kg{3)ka^EAE1BnUHak0=!^R=u>zPDBi@{G*bKpw)g z(tHLZ6E*B$3SIdw8+lor&sRMcxPyareDY7eoxb#4?v*?B^3S&|sBb$Tee|pFUrPx{ zzNa~WgP&qV8rbzqI6R%v{GvxG-lmtIY{;=349g>&~hFECTUAl0D%^}qZJ?|F=Jr3Sp@C%+IJI6f_Uky3;5wZEC< zBN8MM5_DislP5{~|4}Q@8>&+mkOGwRgcQ?xFIW)}|JSb7fNLd688W?_K_h2Xsi^cY z7|G~n#!m_(elOj4^Q`h6Lqja9-y9yuOc*711qfFduKYJ!{jjHydff=LDt_h`El0;lMJ0j3;Ebg`o1Qc=20wat*~Rly zzKHMzQ5ICkqj@g#;X=C|PL3|i$s{STv?ij-X(UY2m!E5_qT&Q@*pZrJ|G&A^K5iwtDS0JEYz(g?Zzg1xC>r&V|-V=xL)+TZnaz*4#%}1k=a^pdGp{ zUv13}QPe*47$`tNO_SXr6(-!diP|lJ?T2YFm;m+`wOZxEK@- zI~A3mDk>4+yN^Hg>~gB%PQv<4507;tLw|@^I|>g9TN$~-V`+Rd_m4SfK<#*bIn@p8 zb~06Vz*i=$DDRd;fE-M)KPMIHBybrx1%28< ziFTe=QON}U{zhy6-pAWZBPuEvB_xJTsDrYx4Mc9x62^F?vb5Tk4gOQ+FhU{la>{>K zQW{J2Uh>9%FT_LVDNHF&&DPekl4G3hX{{2EZ#qe#HmHY(n2}*$m(7wDbyYG3Y&p%x z%n}|(S&?T!_M?4H;P3{Kz9qVh4j;k;Z%lqd!dsCfslL^skbS!|kPfu9+QTl+C1iT^ zo=HU{1)F1ACAUBB#w;G;2>>_o12^K&!!F&2h-kxnE>4bwxvNJJq#j(e1ie7Nt$Y=b zrL)Ma@sYOwIn?0lEQtr?|B{_Sh5u7e<{To~_u?H`m*oG)q#$Jng~Ue@2LIshVuh8H zX;+e&A~AA6?4R9Th@X()sPr%}B}Egy*#v;jDrWyjP`ed!_}nDKa&NQyA3=D61=O69 zOqb7#%RyrfUDd>YLO|lynsW5& z4kW)sVaOH8!Jw#a$9vkOZ0juvd)f|VZ5b4z?6ajDirt|uZd=5W2OW@iMcf0Aiz3d* zqXyW;&Xwrm%9-u#QGE;2%iyDeqTAqEp+e|-_IW35j3Vy0R+jG8Wd(FXU`ax~Dus!l z*TlGwVotWRD#>pw)yOOvL?VxYdn=?0PV65#BZXREUh|R?f`v{2?Cc9JTv*(PavWG`rLc%QX+0~fnh~ex`fp$=*D(gKXF~k30mjfx`$ij=6(~P&u zqyNaHn*z@dGw_2h&jt)dPl}E-B-lIDRz1#C{6QP};J)agv<5cKWzJk693A|4z53Eh;iXpl(gP*q%mQy$?*VPQZP_ZB zPhxWRu77`;nNF;tR;@4w{t|tDDyY-mWO8b=*XO^HP{V0R)R7FCAFvUlU4`=mfcNeGG~1u99y zkOwJH1zY@r6u1`~wshFArNf5*r;{YRoH574uyoZ(mN5R38U4A63W;=$b^oLXZ_Dt| zKk2Z5&sqj^B5*Rzx@zmd?SVZG&oTC>*g*;afi0!u>EvYA{0lz&}~L zGfd_>@a7trWjnsJA((`n>KEiZLMid9hUDEnVg_FXsJ>y_pAE80_Yrop+X&uJy|goC654-a7MC&dFS)% ztBStZ;r?1{0YAlYrN2=4I8Pk#RBtyw<%;Y|e}=sx4rLX4Bow7&nmK|Fh~2)H9v`NFL(!i*uMSA(JxPzh<3<#YW1THOJ9vBd z&}{}wJ0H8rFm!iDrTTSCs>V6N88_@Mz|jo@uj~=e7W~$&5@)3uYs!`6mFogs|ECit zWy=VL1Wue#1mK_U-1fJnztXj=vBOd2lUH)yKvmTGwa0{9~`hz zxY!z9%=E#QEH33^zjnUB;Ll9$i#x!V(xUn$#5`{gZ{CU9I^H`#qxF!+=4%8S${+0& zQWaU5JO9^{Csb97;zOU#EoanIlvDG5{iB)Wgr2t8cr^QM<;I&@0R}H7D{;)0X%Bi( zdtxtL0~zj^)qLpfarEhz(0SZs@W28h_jAzVi}~}0-gI}DQ`_M7*(b3@*N0Wp3h>y& zK3!YS0;F70QF(L9F{oQS9^?UO-b&dQAN=($-%)z|7;mJHv)DawxIx%MXZwJwX0d>^ zUZ3hda9FQyY~?=se(;svm*i)I_2)G;$o3{LMe18mee&=Kw~{UW*f}!aeroBmBP)!u z-RmeH*l}&36498ZZOn)fRKY^24pez85UTY*&HDa`Qg^h+f)1 zni#@^TYT-8uEj7$^e!LE87e&`v=Gbc-M;x?q{*;DI@4B*Uzyd$G{G&mUQX#XLb#4| zyt|(86wb7DUH?^lkhQ%6_OsO7-RPfs9M}o_mD(Tk0jC@IKS!PjebNg3yL9D0C%;T4 z8sQF=5b82Z&qvqYvJK{%i5hSppGfMcXltF4nbeKtX~v0-@re_KNm+*x9%sXAvmTC4 zV2rH{(tH1*?JnZIeHo}K`DI^sh`A&1=Nnz=Qj)Dh?@>N@UH2J!9uP959R4~~WUf+e zf#W6#-JQyK-aGJH`F z60uv~v7hYwc0#BBSxipcX+#^_TT5(!tgziRyQ@!nJlQN2$8FE)u8dDVciYh3{*UuO zS>qx3Dy2wsxe3kZG*<53TfSU-k@C}yGXs*%q?ewvGWy>Q~AAa^#wNI5kqZ#(dOQA^IkRD^sXD%7ryY^ zZiY{yxaom?jDw|}6k_A`PmTTUh(hAxu1`cLV z%GKYYmvTt*%(}xn*B2BE+;#ca;>WEI-W+)xAnW=Nj(>>1IF|pllnu7of9DB|Jw(AC P#sMr$Y>g`n>2d!b!VCd- literal 0 HcmV?d00001 diff --git a/requirements.txt b/requirements.txt index 98de25159..4858270e6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,2 +1,3 @@ Sphinx==3.5.4 -python-docs-theme==2021.5 +furo +sphinx_copybutton From 63438691a414a1110e45f430563db2e0912aae30 Mon Sep 17 00:00:00 2001 From: Erlend Egeberg Aasland Date: Fri, 4 Jun 2021 20:57:25 +0200 Subject: [PATCH 040/793] Replace Zulip with Discourse (#670) --- communication.rst | 14 ++++++++++---- triaging.rst | 4 ++-- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/communication.rst b/communication.rst index 7d39972ef..90e278e65 100644 --- a/communication.rst +++ b/communication.rst @@ -74,11 +74,17 @@ RSS feed readers. .. _StackOverflow: https://stackoverflow.com/ .. _Libera.Chat: https://libera.chat/ -Zulip ------ -We have our own `zulipchat `_ instance. This should be -used to discuss the development of Python only. +Discourse +--------- + +We have our own `Discourse`_ forum for both developers and users. This forum +complements the `python-dev`_, `python-ideas`_, `python-help`_, and +`python-list`_ mailing lists. Also, voting for new core developers takes place +at `Discourse`_. + +.. _Discourse: https://discuss.python.org/ + IRC --- diff --git a/triaging.rst b/triaging.rst index 4724c9f9c..6c7468b18 100644 --- a/triaging.rst +++ b/triaging.rst @@ -66,13 +66,13 @@ They can request this to any core developer, and the core developer can pass the request to the `Python organization admin `_ on GitHub. The request -can be made confidentially via a DM in Zulip or Discourse, or publicly by opening +can be made confidentially via a DM in Discourse, or publicly by opening an `issue in the core-workflow repository `_. Any contributor who is not already a developer on b.p.o can also self-nominate to be a member of Python triage team. They can request this to any core developer, -confidentially via DM in Zulip or Discourse, or publicly by opening an issue in core-workflow. +confidentially via DM in Discourse, or publicly by opening an issue in core-workflow. If a core developer agrees and is willing to vouch for them, the core developer can pass the request to the GitHub administrator. They should also be added as developer on bpo. From df3ee1a8db42a4292c807db2e9a9e87867a1f8b1 Mon Sep 17 00:00:00 2001 From: Erlend Egeberg Aasland Date: Fri, 4 Jun 2021 20:58:10 +0200 Subject: [PATCH 041/793] Sync section 25.7 with Python/compile.c (#672) --- compiler.rst | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/compiler.rst b/compiler.rst index 9cc2a1644..baa0ae825 100644 --- a/compiler.rst +++ b/compiler.rst @@ -367,6 +367,12 @@ Emission of bytecode is handled by the following macros: ``ADDOP(struct compiler *, int)`` add a specified opcode +``ADDOP_NOLINE(struct compiler *, int)`` + like ``ADDOP`` without a line number; used for artificial opcodes without + no corresponding token in the source code +``ADDOP_IN_SCOPE(struct compiler *, int)`` + like ``ADDOP``, but also exits current scope; used for adding return value + opcodes in lambdas and closures ``ADDOP_I(struct compiler *, int, Py_ssize_t)`` add an opcode that takes an integer argument ``ADDOP_O(struct compiler *, int, PyObject *, TYPE)`` @@ -387,10 +393,14 @@ Emission of bytecode is handled by the following macros: position of the specified PyObject in the consts table. ``ADDOP_LOAD_CONST_NEW(struct compiler *, PyObject *)`` just like ``ADDOP_LOAD_CONST_NEW``, but steals a reference to PyObject -``ADDOP_JABS(struct compiler *, int, basicblock *)`` - create an absolute jump to a basic block -``ADDOP_JREL(struct compiler *, int, basicblock *)`` - create a relative jump to a basic block +``ADDOP_JUMP(struct compiler *, int, basicblock *)`` + create a jump to a basic block +``ADDOP_JUMP_NOLINE(struct compiler *, int, basicblock *)`` + like ``ADDOP_JUMP`` without a line number; used for artificial jumps + without no corresponding token in the source code. +``ADDOP_JUMP_COMPARE(struct compiler *, cmpop_ty)`` + depending on the second argument, add an ``ADDOP_I`` with either an + ``IS_OP``, ``CONTAINS_OP``, or ``COMPARE_OP`` opcode. Several helper functions that will emit bytecode and are named :samp:`compiler_{xx}()` where *xx* is what the function helps with (``list``, From c8079d7cfe38daff9e8d71a4c5313a6591875114 Mon Sep 17 00:00:00 2001 From: andrei kulakov Date: Tue, 8 Jun 2021 17:18:22 -0400 Subject: [PATCH 042/793] Remove superfluous content listing (#712) --- gitbootcamp.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/gitbootcamp.rst b/gitbootcamp.rst index d77a52762..11544f65a 100644 --- a/gitbootcamp.rst +++ b/gitbootcamp.rst @@ -26,8 +26,6 @@ relevant to CPython's workflow. get more information about that in `git documentation `_ -.. contents:: - .. _fork-cpython: Forking CPython GitHub Repository From 58f03637f180039c39c5474fc068fd18e12eb2f3 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Tue, 15 Jun 2021 22:23:30 +0100 Subject: [PATCH 043/793] Restrict the version of sphinx-copybutton to pick up bugfixes (#715) --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 4858270e6..75e94998f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ Sphinx==3.5.4 furo -sphinx_copybutton +sphinx_copybutton>=0.3.3 From 4e6c76a8deb5aa651327f7cb76ee96dd3b73586b Mon Sep 17 00:00:00 2001 From: Harry Date: Mon, 21 Jun 2021 22:16:23 +0100 Subject: [PATCH 044/793] Fix capitalization of PCbuild (GH-719) --- setup.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.rst b/setup.rst index 1a6dde4df..535cdcd2f 100644 --- a/setup.rst +++ b/setup.rst @@ -258,9 +258,9 @@ are downloaded: .. code-block:: dosbatch - PCBuild\build.bat + PCbuild\build.bat -After this build succeeds, you can open the ``PCBuild\pcbuild.sln`` solution in +After this build succeeds, you can open the ``PCbuild\pcbuild.sln`` solution in Visual Studio to continue development. See the `readme`_ for more details on what other software is necessary and how From dd6dba2ca552f2324345cac3350b9ccbf915d524 Mon Sep 17 00:00:00 2001 From: Marco Ippolito Date: Wed, 23 Jun 2021 16:08:25 -0300 Subject: [PATCH 045/793] Add advice on building all modules on Debian-like (GH-673) --- setup.rst | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/setup.rst b/setup.rst index 535cdcd2f..24b7c97d2 100644 --- a/setup.rst +++ b/setup.rst @@ -325,10 +325,14 @@ Then you should update the packages index:: Now you can install the build dependencies via ``apt``:: - $ sudo apt-get build-dep python3.6 + $ sudo apt-get build-dep python3 -If that package is not available for your system, try reducing the minor -version until you find a package that is available. +If you want to build all optional modules, install the following packages and +their dependencies:: + + $ sudo apt-get install build-essential gdb lcov libbz2-dev libffi-dev \ + libgdbm-dev liblzma-dev libncurses5-dev libreadline6-dev \ + libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev .. _MacOS: From 91893b0246ded3e599f4f876d149d44fe3bb58c6 Mon Sep 17 00:00:00 2001 From: Will Schlitzer Date: Thu, 24 Jun 2021 10:46:42 +0100 Subject: [PATCH 046/793] Fix minor typo in coredev.rst (GH-720) This pull request changed "requied" to "required" in the "Gaining Commit Privileges" section in coredev.rst. --- coredev.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/coredev.rst b/coredev.rst index 195b5f559..1066ae2a0 100644 --- a/coredev.rst +++ b/coredev.rst @@ -70,7 +70,7 @@ The steps to gaining commit privileges are: 2. The poll is announced on python-committers 3. Wait for the poll to close and see if the results confirm your membership - as per the voting results requied by PEP 13 + as per the voting results required by PEP 13 4. The person who nominated you emails the steering council with your email address and a request that the council either accept or reject the proposed membership From 0ae9c851419d5eba1af2623129ffd415d75be73c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 14 Jul 2021 18:22:42 -0700 Subject: [PATCH 047/793] Bump sphinx from 3.5.4 to 4.1.1 (#725) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 3.5.4 to 4.1.1. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v3.5.4...v4.1.1) --- updated-dependencies: - dependency-name: sphinx dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 75e94998f..60f42feeb 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ -Sphinx==3.5.4 +Sphinx==4.1.1 furo sphinx_copybutton>=0.3.3 From f8ca097b0eeb1f85863a631044ac2765aea670d2 Mon Sep 17 00:00:00 2001 From: Mariatta Wijaya Date: Thu, 15 Jul 2021 12:11:33 -0700 Subject: [PATCH 048/793] Delete travis.yml file (#726) GitHub Actions seem sufficient for our needs, we don't need both running at the same time. --- .travis.yml | 16 ---------------- 1 file changed, 16 deletions(-) delete mode 100644 .travis.yml diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 5608206df..000000000 --- a/.travis.yml +++ /dev/null @@ -1,16 +0,0 @@ -language: python -python: 3.6 -cache: pip - -install: python3 -m pip install -U pip -r requirements.txt - -jobs: - include: - - stage: build-docs - script: sphinx-build -n -W -q -b html -d _build/doctrees . _build/html - - - stage: link-check - script: sphinx-build -b linkcheck -d _build/doctrees . _build/linkcheck - - allow_failures: - - stage: link-check From 0e6f02833f0adb4a229bacf425b56a3620a50474 Mon Sep 17 00:00:00 2001 From: Edison J Abahurire <20975616+SimiCode@users.noreply.github.com> Date: Fri, 16 Jul 2021 00:20:56 +0300 Subject: [PATCH 049/793] Fixes #717: Example for Running a single test case fails (#718) The class seems to be reloaded using a different name at the bottom of `https://github.com/python/cpython/blob/main/Lib/test/test_abc.py` --- runtests.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/runtests.rst b/runtests.rst index 74e38ff8b..2d6c3eb48 100644 --- a/runtests.rst +++ b/runtests.rst @@ -48,7 +48,7 @@ verbose mode (using ``-v``), so that individual failures are detailed:: To run a single test case, use the ``unittest`` module, providing the import path to the test case:: - ./python -m unittest -v test.test_abc.TestABC + ./python -m unittest -v test.test_abc.TestABC_Py If you have a multi-core or multi-CPU machine, you can enable parallel testing using several Python processes so as to speed up things:: From af52f4be7c9fab2ec2b658b130fc1686c3586b04 Mon Sep 17 00:00:00 2001 From: Dennis Sweeney <36520290+sweeneyde@users.noreply.github.com> Date: Fri, 16 Jul 2021 18:03:18 -0400 Subject: [PATCH 050/793] bpo-42349: Clarify basicblocks and give some examples (#714) Co-authored-by: Mariatta Wijaya --- compiler.rst | 56 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 20 deletions(-) diff --git a/compiler.rst b/compiler.rst index baa0ae825..91fe254d2 100644 --- a/compiler.rst +++ b/compiler.rst @@ -288,26 +288,42 @@ number is passed as the last parameter to each ``stmt_ty`` function. Control Flow Graphs ------------------- -A control flow graph (often referenced by its acronym, CFG) is a -directed graph that models the flow of a program using basic blocks that -contain the intermediate representation (abbreviated "IR", and in this -case is Python bytecode) within the blocks. Basic blocks themselves are -a block of IR that has a single entry point but possibly multiple exit -points. The single entry point is the key to basic blocks; it all has -to do with jumps. An entry point is the target of something that -changes control flow (such as a function call or a jump) while exit -points are instructions that would change the flow of the program (such -as jumps and 'return' statements). What this means is that a basic -block is a chunk of code that starts at the entry point and runs to an -exit point or the end of the block. - -As an example, consider an 'if' statement with an 'else' block. The -guard on the 'if' is a basic block which is pointed to by the basic -block containing the code leading to the 'if' statement. The 'if' -statement block contains jumps (which are exit points) to the true body -of the 'if' and the 'else' body (which may be ``NULL``), each of which are -their own basic blocks. Both of those blocks in turn point to the -basic block representing the code following the entire 'if' statement. +A *control flow graph* (often referenced by its acronym, CFG) is a +directed graph that models the flow of a program. A node of a CFG is +not an individual bytecode instruction, but instead represents a +sequence of bytecode instructions that always execute sequentially. +Each node is called a *basic block* and must always execute from +start to finish, with a single entry point at the beginning and a +single exit point at the end. If some bytecode instruction *a* needs +to jump to some other bytecode instruction *b*, then *a* must occur at +the end of its basic block, and *b* must occur at the start of its +basic block. + +As an example, consider the following code snippet: + +.. code-block:: Python + + if x < 10: + f1() + f2() + else: + g() + end() + +The ``x < 10`` guard is represented by its own basic block that +compares ``x`` with ``10`` and then ends in a conditional jump based on +the result of the comparison. This conditional jump allows the block +to point to both the body of the ``if`` and the body of the ``else``. The +``if`` basic block contains the ``f1()`` and ``f2()`` calls and points to +the ``end()`` basic block. The ``else`` basic block contains the ``g()`` +call and similarly points to the ``end()`` block. + +Note that more complex code in the guard, the ``if`` body, or the ``else`` +body may be represented by multiple basic blocks. For instance, +short-circuiting boolean logic in a guard like ``if x or y:`` +will produce one basic block that tests the truth value of ``x`` +and then points both (1) to the start of the ``if`` body and (2) to +a different basic block that tests the truth value of y. CFGs are usually one step away from final code output. Code is directly generated from the basic blocks (with jump targets adjusted based on the From 37c53440bc0b504dc8a54486ca2c5fb0daed338f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Sat, 17 Jul 2021 10:26:28 +0200 Subject: [PATCH 051/793] Replace @ilevkivskyi with @Fidget-Spinner as typing expert (#728) --- experts.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/experts.rst b/experts.rst index 001ff642a..dc84bdfb9 100644 --- a/experts.rst +++ b/experts.rst @@ -241,7 +241,7 @@ tracemalloc vstinner tty twouters* turtle gregorlingl, willingc types yselivanov -typing gvanrossum, levkivskyi* +typing gvanrossum, kj unicodedata lemburg, ezio.melotti unittest michael.foord*, ezio.melotti, rbcollins unittest.mock michael.foord* From 69c49e1b8d17c2f3e6fe3f4c22a0d78e6702a7f6 Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Fri, 30 Jul 2021 23:15:13 +0800 Subject: [PATCH 052/793] Add build.bat counterparts to to checklist (GH-729) --- grammar.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/grammar.rst b/grammar.rst index 73226e115..571d6869c 100644 --- a/grammar.rst +++ b/grammar.rst @@ -23,14 +23,16 @@ Checklist Note: sometimes things mysteriously don't work. Before giving up, try ``make clean``. * :file:`Grammar/python.gram`: The grammar, with actions that build AST nodes. After changing - it, run ``make regen-pegen``, to regenerate :file:`Parser/parser.c`. + it, run ``make regen-pegen`` (or ``build.bat --regen`` on Windows), to + regenerate :file:`Parser/parser.c`. (This runs Python's parser generator, ``Tools/peg_generator``). * :file:`Grammar/Tokens` is a place for adding new token types. After changing it, run ``make regen-token`` to regenerate :file:`Include/token.h`, :file:`Parser/token.c`, :file:`Lib/token.py` and :file:`Doc/library/token-list.inc`. If you change both ``python.gram`` and ``Tokens``, - run ``make regen-token`` before ``make regen-pegen``. + run ``make regen-token`` before ``make regen-pegen``. On Windows, + ``build.bat --regen`` will regenerate both at the same time. * :file:`Parser/Python.asdl` may need changes to match the grammar. Then run ``make regen-ast`` to regenerate :file:`Include/Python-ast.h` and :file:`Python/Python-ast.c`. From 8d97b43d87477a9c7fba439a9a0916e18be97fd8 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 30 Jul 2021 17:16:12 +0200 Subject: [PATCH 053/793] Bump sphinx from 4.1.1 to 4.1.2 (GH-732) Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 4.1.1 to 4.1.2. - [Release notes](https://github.com/sphinx-doc/sphinx/releases) - [Changelog](https://github.com/sphinx-doc/sphinx/blob/4.x/CHANGES) - [Commits](https://github.com/sphinx-doc/sphinx/compare/v4.1.1...v4.1.2) Signed-off-by: dependabot[bot] --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 60f42feeb..c90e30088 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ -Sphinx==4.1.1 +Sphinx==4.1.2 furo sphinx_copybutton>=0.3.3 From 187740f715c036de61e4330a47f2ee406d06a335 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Fri, 30 Jul 2021 17:19:06 +0200 Subject: [PATCH 054/793] bpo-44777: Explicitly list the python-buildbots mailing list as a contact method (GH-733) --- buildbots.rst | 22 ++++++++++++++++++++++ pullrequest.rst | 3 +++ 2 files changed, 25 insertions(+) diff --git a/buildbots.rst b/buildbots.rst index ca342938d..47f319c30 100644 --- a/buildbots.rst +++ b/buildbots.rst @@ -25,6 +25,28 @@ build results after you push a change to the repository. It is therefore important that you get acquainted with the way these results are presented, and how various kinds of failures can be explained and diagnosed. +In case of trouble +------------------ + +Please read this page in full. If your questions aren't answered here and you +need assistance with the buildbots, a good way to get help is to either: + +* contact the ``python-buildbots@python.org`` mailing list where all buildbot + worker owners are subscribed; or +* contact the release manager of the branch you have issues with. + +Buildbot failures on Pull Requests +---------------------------------- + +The ``bedevere-bot`` on GitHub will put a message on your merged Pull Request +if building your commit on a stable buildbot worker fails. Take care to +evaluate the failure, even if it looks unrelated at first glance. + +Not all failures will generate a notification since not all builds are executed +after each commit. In particular, reference leaks builds take several hours to +complete so they are done periodically. This is why it's important for you to +be able to check the results yourself, too. + Checking results of automatic builds ------------------------------------ diff --git a/pullrequest.rst b/pullrequest.rst index 7cb194490..e6b6b767c 100644 --- a/pullrequest.rst +++ b/pullrequest.rst @@ -37,6 +37,9 @@ Here is a quick overview of how you can contribute to CPython: #. `Create Pull Request`_ on GitHub to merge a branch from your fork +#. Make sure the continuous integration checks on your Pull Request + are green (i.e. successful) + #. Review and address `comments on your Pull Request`_ #. When your changes are merged, you can :ref:`delete the PR branch From 4edf354b82aece8c6b7bc0c5fb2b082a06071f5d Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Wed, 4 Aug 2021 17:54:48 +0100 Subject: [PATCH 055/793] Add a new guide on how to understand and use the PEG parser (#734) --- compiler.rst | 1 + grammar.rst | 16 +- index.rst | 3 +- parser.rst | 915 +++++++++++++++++++++++++++++++++++++++++++++++ requirements.txt | 2 +- 5 files changed, 928 insertions(+), 9 deletions(-) create mode 100644 parser.rst diff --git a/compiler.rst b/compiler.rst index 91fe254d2..85943118b 100644 --- a/compiler.rst +++ b/compiler.rst @@ -40,6 +40,7 @@ these (see :doc:`grammar`). Abstract Syntax Trees (AST) --------------------------- +.. _compiler-ast-trees: .. sidebar:: Green Tree Snakes diff --git a/grammar.rst b/grammar.rst index 571d6869c..471d4e827 100644 --- a/grammar.rst +++ b/grammar.rst @@ -9,13 +9,15 @@ Abstract There's more to changing Python's grammar than editing :file:`Grammar/python.gram`. Here's a checklist. -NOTE: These instructions are for Python 3.9 and beyond. Earlier -versions use a different parser technology. You probably shouldn't -try to change the grammar of earlier Python versions, but if you -really want to, use GitHub to track down the earlier version of this -file in the devguide. (Python 3.9 itself actually supports both -parsers; the old parser can be invoked by passing ``-X oldparser``.) - +.. note:: + These instructions are for Python 3.9 and beyond. Earlier + versions use a different parser technology. You probably shouldn't + try to change the grammar of earlier Python versions, but if you + really want to, use GitHub to track down the earlier version of this + file in the devguide. + +For more information on how to use the new parser, check the +:ref:`section on how to use CPython's parser `. Checklist --------- diff --git a/index.rst b/index.rst index 7c7810ca8..37afb6e85 100644 --- a/index.rst +++ b/index.rst @@ -262,6 +262,7 @@ Additional Resources * Help with ... * :doc:`exploring` * :doc:`grammar` + * :doc:`parser` * :doc:`compiler` * :doc:`garbage_collector` * Tool support @@ -293,7 +294,6 @@ Full Table of Contents ---------------------- .. toctree:: - :numbered: :maxdepth: 3 setup @@ -320,6 +320,7 @@ Full Table of Contents gdb exploring grammar + parser compiler garbage_collector extensions diff --git a/parser.rst b/parser.rst new file mode 100644 index 000000000..71387b639 --- /dev/null +++ b/parser.rst @@ -0,0 +1,915 @@ +.. _parser: + +Guide of CPython's Parser +========================= + +:Author: Pablo Galindo Salgado + +.. highlight:: none + +Abstract +-------- + +The Parser in CPython is currently a `PEG (Parser Expression Grammar) +`_ parser. The first +version of the parser used to be an `LL(1) +`_ based parser that was one of the +oldest parts of CPython implemented before it was replaced by :pep:`617`. In +particular, both the current parser and the old LL(1) parser are the output of a +`parser generator `_. This +means that the way the parser is written is by feeding a description of the +Grammar of the Python language to a special program (the parser generator) which +outputs the parser. The way the Python language is changed is therefore by +modifying the grammar file and developers rarely need to interact with the +parser generator itself other than use it to generate the parser. + +How PEG Parsers Work +-------------------- + +.. _how-peg-parsers-work: + +A PEG (Parsing Expression Grammar) grammar (like the current one) differs from a +context-free grammar in that the way it is written more closely +reflects how the parser will operate when parsing it. The fundamental technical +difference is that the choice operator is ordered. This means that when writing:: + + rule: A | B | C + +a context-free-grammar parser (like an LL(1) parser) will generate constructions +that given an input string will *deduce* which alternative (``A``, ``B`` or ``C``) +must be expanded, while a PEG parser will check if the first alternative succeeds +and only if it fails, will it continue with the second or the third one in the +order in which they are written. This makes the choice operator not commutative. + +Unlike LL(1) parsers, PEG-based parsers cannot be ambiguous: if a string parses, +it has exactly one valid parse tree. This means that a PEG-based parser cannot +suffer from the ambiguity problems that can arise with LL(1) parsers and with +context-free grammars in general. + +PEG parsers are usually constructed as a recursive descent parser in which every +rule in the grammar corresponds to a function in the program implementing the +parser and the parsing expression (the "expansion" or "definition" of the rule) +represents the "code" in said function. Each parsing function conceptually takes +an input string as its argument, and yields one of the following results: + +* A "success" result. This result indicates that the expression can be parsed by + that rule and the function may optionally move forward or consume one or more + characters of the input string supplied to it. +* A "failure" result, in which case no input is consumed. + +Notice that "failure" results do not imply that the program is incorrect, nor do +they necessarily mean that the parsing has failed. Since the choice operator is +ordered, a failure very often merely indicates "try the following option". A +direct implementation of a PEG parser as a recursive descent parser will present +exponential time performance in the worst case, because PEG parsers have +infinite lookahead (this means that they can consider an arbitrary number of +tokens before deciding for a rule). Usually, PEG parsers avoid this exponential +time complexity with a technique called "packrat parsing" [1]_ which not only +loads the entire program in memory before parsing it but also allows the parser +to backtrack arbitrarily. This is made efficient by memoizing the rules already +matched for each position. The cost of the memoization cache is that the parser +will naturally use more memory than a simple LL(1) parser, which normally are +table-based. + + +Key ideas +~~~~~~~~~ + +.. important:: + Don't try to reason about a PEG grammar in the same way you would to with an EBNF + or context free grammar. PEG is optimized to describe **how** input strings will + be parsed, while context-free grammars are optimized to generate strings of the + language they describe (in EBNF, to know if a given string is in the language, you need + to do work to find out as it is not immediately obvious from the grammar). + +* Alternatives are ordered ( ``A | B`` is not the same as ``B | A`` ). +* If a rule returns a failure, it doesn't mean that the parsing has failed, + it just means "try something else". +* By default PEG parsers run in exponential time, which can be optimized to linear by + using memoization. +* If parsing fails completely (no rule succeeds in parsing all the input text), the + PEG parser doesn't have a concept of "where the :exc:`SyntaxError` is". + + +Consequences or the ordered choice operator +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. _consequences-of-ordered-choice: + +Although PEG may look like EBNF, its meaning is quite different. The fact +that in PEG parsers alternatives are ordered (which is at the core of how PEG +parsers work) has deep consequences, other than removing ambiguity. + +If a rule has two alternatives and the first of them succeeds, the second one is +**not** attempted even if the caller rule fails to parse the rest of the input. +Thus the parser is said to be "eager". To illustrate this, consider +the following two rules (in these examples, a token is an individual character): :: + + first_rule: ( 'a' | 'aa' ) 'a' + second_rule: ('aa' | 'a' ) 'a' + +In a regular EBNF grammar, both rules specify the language ``{aa, aaa}`` but +in PEG, one of these two rules accepts the string ``aaa`` but not the string +``aa``. The other does the opposite -- it accepts the string the string ``aa`` +but not the string ``aaa``. The rule ``('a'|'aa')'a'`` does +not accept ``aaa`` because ``'a'|'aa'`` consumes the first ``a``, letting the +final ``a`` in the rule consume the second, and leaving out the third ``a``. +As the rule has succeeded, no attempt is ever made to go back and let +``'a'|'aa'`` try the second alternative. The expression ``('aa'|'a')'a'`` does +not accept ``aa`` because ``'aa'|'a'`` accepts all of ``aa``, leaving nothing +for the final ``a``. Again, the second alternative of ``'aa'|'a'`` is not +tried. + +.. caution:: + + The effects of ordered choice, such as the ones illustrated above, may be hidden by many levels of rules. + +For this reason, writing rules where an alternative is contained in the next one is in almost all cases a mistake, +for example: :: + + my_rule: + | 'if' expression 'then' block + | 'if' expression 'then' block 'else' block + +In this example, the second alternative will never be tried because the first one will +succeed first (even if the input string has an ``'else' block`` that follows). To correctly +write this rule you can simply alter the order: :: + + my_rule: + | 'if' expression 'then' block 'else' block + | 'if' expression 'then' block + +In this case, if the input string doesn't have an ``'else' block``, the first alternative +will fail and the second will be attempted without said part. + +Syntax +------ + +The grammar consists of a sequence of rules of the form: :: + + rule_name: expression + +Optionally, a type can be included right after the rule name, which +specifies the return type of the C or Python function corresponding to +the rule: :: + + rule_name[return_type]: expression + +If the return type is omitted, then a ``void *`` is returned in C and an +``Any`` in Python. + +Grammar Expressions +~~~~~~~~~~~~~~~~~~~ + +``# comment`` +''''''''''''' + +Python-style comments. + +``e1 e2`` +''''''''' + +Match e1, then match e2. + +:: + + rule_name: first_rule second_rule + +``e1 | e2`` +''''''''''' + +Match e1 or e2. + +The first alternative can also appear on the line after the rule name +for formatting purposes. In that case, a \| must be used before the +first alternative, like so: + +:: + + rule_name[return_type]: + | first_alt + | second_alt + +``( e )`` +''''''''' + +Match e. + +:: + + rule_name: (e) + +A slightly more complex and useful example includes using the grouping +operator together with the repeat operators: + +:: + + rule_name: (e1 e2)* + +``[ e ] or e?`` +''''''''''''''' + +Optionally match e. + +:: + + rule_name: [e] + +A more useful example includes defining that a trailing comma is +optional: + +:: + + rule_name: e (',' e)* [','] + +``e*`` +'''''' + +Match zero or more occurrences of e. + +:: + + rule_name: (e1 e2)* + +``e+`` +'''''' + +Match one or more occurrences of e. + +:: + + rule_name: (e1 e2)+ + +``s.e+`` +'''''''' + +Match one or more occurrences of e, separated by s. The generated parse +tree does not include the separator. This is otherwise identical to +``(e (s e)*)``. + +:: + + rule_name: ','.e+ + +``&e`` +'''''' + +.. _peg-positive-lookahead: + +Succeed if e can be parsed, without consuming any input. + +``!e`` +'''''' + +.. _peg-negative-lookahead: + +Fail if e can be parsed, without consuming any input. + +An example taken from the Python grammar specifies that a primary +consists of an atom, which is not followed by a ``.`` or a ``(`` or a +``[``: + +:: + + primary: atom !'.' !'(' !'[' + +``~`` +'''''' + +Commit to the current alternative, even if it fails to parse. + +:: + + rule_name: '(' ~ some_rule ')' | some_alt + +In this example, if a left parenthesis is parsed, then the other +alternative won’t be considered, even if some_rule or ‘)’ fail to be +parsed. + +Left recursion +~~~~~~~~~~~~~~ + +PEG parsers normally do not support left recursion but CPython's parser +generator implements a technique similar to the one described in Medeiros et al. +[2]_ but using the memoization cache instead of static variables. This approach +is closer to the one described in Warth et al. [3]_. This allows us to write not +only simple left-recursive rules but also more complicated rules that involve +indirect left-recursion like:: + + rule1: rule2 | 'a' + rule2: rule3 | 'b' + rule3: rule1 | 'c' + +and "hidden left-recursion" like:: + + rule: 'optional'? rule '@' some_other_rule + +Variables in the Grammar +~~~~~~~~~~~~~~~~~~~~~~~~ + +A sub-expression can be named by preceding it with an identifier and an +``=`` sign. The name can then be used in the action (see below), like this: :: + + rule_name[return_type]: '(' a=some_other_rule ')' { a } + +Grammar actions +~~~~~~~~~~~~~~~ + +.. _peg-grammar-actions: + +To avoid the intermediate steps that obscure the relationship between the +grammar and the AST generation the PEG parser allows directly generating AST +nodes for a rule via grammar actions. Grammar actions are language-specific +expressions that are evaluated when a grammar rule is successfully parsed. These +expressions can be written in Python or C depending on the desired output of the +parser generator. This means that if one would want to generate a parser in +Python and another in C, two grammar files should be written, each one with a +different set of actions, keeping everything else apart from said actions +identical in both files. As an example of a grammar with Python actions, the +piece of the parser generator that parses grammar files is bootstrapped from a +meta-grammar file with Python actions that generate the grammar tree as a result +of the parsing. + +In the specific case of the PEG grammar for Python, having actions allows +directly describing how the AST is composed in the grammar itself, making it +more clear and maintainable. This AST generation process is supported by the use +of some helper functions that factor out common AST object manipulations and +some other required operations that are not directly related to the grammar. + +To indicate these actions each alternative can be followed by the action code +inside curly-braces, which specifies the return value of the alternative:: + + rule_name[return_type]: + | first_alt1 first_alt2 { first_alt1 } + | second_alt1 second_alt2 { second_alt1 } + +If the action is omitted and C code is being generated, then there are two +different possibilities: + +1. If there’s a single name in the alternative, this gets returned. +2. If not, a dummy name object gets returned (this case should be avoided). + +If the action is ommited, a default action is generated: + +* If there's a single name in the rule in the rule, it gets returned. + +* If there is more than one name in the rule, a collection with all parsed + expressions gets returned (the type of the collection will be different + in C and Python). + +This default behaviour is primarily made for very simple situations and for +debugging pourposes. + +The full meta-grammar for the grammars supported by the PEG generator is: + +:: + + start[Grammar]: grammar ENDMARKER { grammar } + + grammar[Grammar]: + | metas rules { Grammar(rules, metas) } + | rules { Grammar(rules, []) } + + metas[MetaList]: + | meta metas { [meta] + metas } + | meta { [meta] } + + meta[MetaTuple]: + | "@" NAME NEWLINE { (name.string, None) } + | "@" a=NAME b=NAME NEWLINE { (a.string, b.string) } + | "@" NAME STRING NEWLINE { (name.string, literal_eval(string.string)) } + + rules[RuleList]: + | rule rules { [rule] + rules } + | rule { [rule] } + + rule[Rule]: + | rulename ":" alts NEWLINE INDENT more_alts DEDENT { + Rule(rulename[0], rulename[1], Rhs(alts.alts + more_alts.alts)) } + | rulename ":" NEWLINE INDENT more_alts DEDENT { Rule(rulename[0], rulename[1], more_alts) } + | rulename ":" alts NEWLINE { Rule(rulename[0], rulename[1], alts) } + + rulename[RuleName]: + | NAME '[' type=NAME '*' ']' {(name.string, type.string+"*")} + | NAME '[' type=NAME ']' {(name.string, type.string)} + | NAME {(name.string, None)} + + alts[Rhs]: + | alt "|" alts { Rhs([alt] + alts.alts)} + | alt { Rhs([alt]) } + + more_alts[Rhs]: + | "|" alts NEWLINE more_alts { Rhs(alts.alts + more_alts.alts) } + | "|" alts NEWLINE { Rhs(alts.alts) } + + alt[Alt]: + | items '$' action { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=action) } + | items '$' { Alt(items + [NamedItem(None, NameLeaf('ENDMARKER'))], action=None) } + | items action { Alt(items, action=action) } + | items { Alt(items, action=None) } + + items[NamedItemList]: + | named_item items { [named_item] + items } + | named_item { [named_item] } + + named_item[NamedItem]: + | NAME '=' ~ item {NamedItem(name.string, item)} + | item {NamedItem(None, item)} + | it=lookahead {NamedItem(None, it)} + + lookahead[LookaheadOrCut]: + | '&' ~ atom {PositiveLookahead(atom)} + | '!' ~ atom {NegativeLookahead(atom)} + | '~' {Cut()} + + item[Item]: + | '[' ~ alts ']' {Opt(alts)} + | atom '?' {Opt(atom)} + | atom '*' {Repeat0(atom)} + | atom '+' {Repeat1(atom)} + | sep=atom '.' node=atom '+' {Gather(sep, node)} + | atom {atom} + + atom[Plain]: + | '(' ~ alts ')' {Group(alts)} + | NAME {NameLeaf(name.string) } + | STRING {StringLeaf(string.string)} + + # Mini-grammar for the actions + + action[str]: "{" ~ target_atoms "}" { target_atoms } + + target_atoms[str]: + | target_atom target_atoms { target_atom + " " + target_atoms } + | target_atom { target_atom } + + target_atom[str]: + | "{" ~ target_atoms "}" { "{" + target_atoms + "}" } + | NAME { name.string } + | NUMBER { number.string } + | STRING { string.string } + | "?" { "?" } + | ":" { ":" } + +As an illustrative example this simple grammar file allows directly +generating a full parser that can parse simple arithmetic expressions and that +returns a valid C-based Python AST: + +:: + + start[mod_ty]: a=expr_stmt* ENDMARKER { _PyAST_Module(a, NULL, p->arena) } + expr_stmt[stmt_ty]: a=expr NEWLINE { _PyAST_Expr(a, EXTRA) } + + expr[expr_ty]: + | l=expr '+' r=term { _PyAST_BinOp(l, Add, r, EXTRA) } + | l=expr '-' r=term { _PyAST_BinOp(l, Sub, r, EXTRA) } + | term + + term[expr_ty]: + | l=term '*' r=factor { _PyAST_BinOp(l, Mult, r, EXTRA) } + | l=term '/' r=factor { _PyAST_BinOp(l, Div, r, EXTRA) } + | factor + + factor[expr_ty]: + | '(' e=expr ')' { e } + | atom + + atom[expr_ty]: + | NAME + | NUMBER + +Here ``EXTRA`` is a macro that expands to ``start_lineno, start_col_offset, +end_lineno, end_col_offset, p->arena``, those being variables automatically +injected by the parser; ``p`` points to an object that holds on to all state +for the parser. + +A similar grammar written to target Python AST objects: + +:: + + start[ast.Module]: a=expr_stmt* ENDMARKER { ast.Module(body=a or [] } + expr_stmt: a=expr NEWLINE { ast.Expr(value=a, EXTRA) } + + expr: + | l=expr '+' r=term { ast.BinOp(left=l, op=ast.Add(), right=r, EXTRA) } + | l=expr '-' r=term { ast.BinOp(left=l, op=ast.Sub(), right=r, EXTRA) } + | term + + term: + | l=term '*' r=factor { ast.BinOp(left=l, op=ast.Mult(), right=r, EXTRA) } + | l=term '/' r=factor { ast.BinOp(left=l, op=ast.Div(), right=r, EXTRA) } + | factor + + factor: + | '(' e=expr ')' { e } + | atom + + atom: + | NAME + | NUMBER + + +Pegen +----- + +Pegen is the parser generator used in CPython to produce the final PEG parser used by the interpreter. It is the +program that can be used to read the python grammar located in :file:`Grammar/Python.gram` and produce the final C +parser. It contains the following pieces: + +* A parser generator that can read a grammar file and produce a PEG parser written in Python or C that can parse + said grammar. The generator is located at :file:`Tools/peg_generator/pegen`. +* A PEG meta-grammar that automatically generates a Python parser that is used for the parser generator itself + (this means that there are no manually-written parsers). The meta-grammar is + located at :file:`Tools/peg_generator/pegen/metagrammar.gram`. +* A generated parser (using the parser generator) that can directly produce C and Python AST objects. + +The source code for Pegen lives at :file:`Tools/peg_generator/pegen` but normally all typical commands to interact +with the parser generator are executed from the main makefile. + +How to regenerate the parser +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Once you have made the changes to the grammar files, to regenerate the ``C`` +parser (the one used by the interpreter) just execute: :: + + make regen-pegen + +using the :file:`Makefile` in the main directory. If you are on Windows you can +use the Visual Studio project files to regenerate the parser or to execute: :: + + ./PCbuild/build.bat --regen + +The generated parser file is located at :file:`Parser/parser.c`. + +How to regenerate the meta-parser +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The meta-grammar (the grammar that describes the grammar for the grammar files +themselves) is located at :file:`Tools/peg_generator/pegen/metagrammar.gram`. +Although it is very unlikely that you will ever need to modify it, if you make any modifications +to this file (in order to implement new Pegen features) you will need to regenerate +the meta-parser (the parser that parses the grammar files). To do so just execute: :: + + make regen-pegen-metaparser + +If you are on Windows you can use the Visual Studio project files +to regenerate the parser or to execute: :: + + ./PCbuild/build.bat --regen + + +Grammatical elements and rules +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Pegen has some special grammatical elements and rules: + +* Strings with single quotes (') (e.g. ``'class'``) denote KEYWORDS. +* Strings with double quotes (") (e.g. ``"match"``) denote SOFT KEYWORDS. +* Upper case names (e.g. ``NAME``) denote tokens in the :file:`Grammar/Tokens` file. +* Rule names starting with `invalid_` are used for specialized syntax errors. + + - These rules are NOT used in the first pass of the parser. + - Only if the first pass fails to parse, a second pass including the invalid + rules will be executed. + - If the parser fails in the second phase with a generic syntax error, the + location of the generic failure of the first pass will be used (this avoids + reporting incorrect locations due to the invalid rules). + - The order of the alternatives involving invalid rules matter + (like any rule in PEG). + +Tokenization +~~~~~~~~~~~~ + +It is common among PEG parser frameworks that the parser does both the parsing and the tokenization, +but this does not happen in Pegen. The reason is that the Python language needs a custom tokenizer +to handle things like indentation boundaries, some special keywords like ``ASYNC`` and ``AWAIT`` +(for compatibility purposes), backtracking errors (such as unclosed parenthesis), dealing with encoding, +interactive mode and much more. Some of these reasons are also there for historical purposes, and some +others are useful even today. + +The list of tokens (all uppercase names in the grammar) that you can use can be found in the :file:`Grammar/Tokens` +file. If you change this file to add new tokens, make sure to regenerate the files by executing: :: + + make regen-token + +If you are on Windows you can use the Visual Studio project files to regenerate the tokens or to execute: :: + + ./PCbuild/build.bat --regen + +How tokens are generated and the rules governing this is completely up to the tokenizer (:file:`Parser/tokenizer.c`) +and the parser just receives tokens from it. + +Memoization +~~~~~~~~~~~ + +As described previously, to avoid exponential time complexity in the parser, memoization is used. + +The C parser used by Python is highly optimized and memoization can be expensive both in memory and time. Although +the memory cost is obvious (the parser needs memory for storing previous results in the cache) the execution time +cost comes for continuously checking if the given rule has a cache hit or not. In many situations, just parsing it +again can be faster. Pegen **disables memoization by default** except for rules with the special marker `memo` after +the rule name (and type, if present): :: + + rule_name[typr] (memo): + ... + +By selectively turning on memoization for a handful of rules, the parser becomes faster and uses less memory. + +.. note:: + Left-recursive rules always use memoization, since the implementation of left-recursion depends on it. + +To know if a new rule needs memoization or not, benchmarking is required +(comparing execution times and memory usage of some considerably big files with +and without memoization). There is a very simple instrumentation API available +in the generated C parse code that allows to measure how much each rule uses +memoization (check the :file:`Parser/pegen.c` file for more information) but it +needs to be manually activated. + +Automatic variables +~~~~~~~~~~~~~~~~~~~ + +To make writing actions easier, Pegen injects some automatic variables in the namespace available +when writing actions. In the C parser, some of these automatic variable names are: + +* ``p``: The parser structure. +* ``EXTRA``: This is a macro that expands to ``(_start_lineno, _start_col_offset, _end_lineno, _end_col_offset, p->arena)``, + which is normally used to create AST nodes as almost all constructors need these attributes to be provided. All of the + location variables are taken from the location information of the current token. + +Hard and Soft keywords +~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + In the grammar files, keywords are defined using **single quotes** (e.g. `'class'`) while soft + keywords are defined using **double quotes** (e.g. `"match"`). + +There are two kinds of keywords allowed in pegen grammars: *hard* and *soft* +keywords. The difference between hard and soft keywords is that hard keywords +are always reserved words, even in positions where they make no sense (e.g. ``x = class + 1``), +while soft keywords only get a special meaning in context. Trying to use a hard +keyword as a variable will always fail: + +.. code-block:: + + >>> class = 3 + File "", line 1 + class = 3 + ^ + SyntaxError: invalid syntax + >>> foo(class=3) + File "", line 1 + foo(class=3) + ^^^^^ + SyntaxError: invalid syntax + +While soft keywords don't have this limitation if used in a context other the one where they +are defined as keywords: + +.. code-block:: python + + >>> match = 45 + >>> foo(match="Yeah!") + +The ``match`` and ``case`` keywords are soft keywords, so that they are recognized as +keywords at the beginning of a match statement or case block respectively, but are +allowed to be used in other places as variable or argument names. + +You can get a list of all keywords defined in the grammar from Python: + +.. code-block:: python + + >>> import keyword + >>> keyword.kwlist + ['False', 'None', 'True', 'and', 'as', 'assert', 'async', 'await', 'break', + 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', + 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', + 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield'] + +as well as soft keywords: + +.. code-block:: python + + >>> import keyword + >>> keyword.softkwlist + ['_', 'case', 'match'] + +.. caution:: + Soft keywords can be a bit challenging to manage as they can be accepted in + places you don't intend to, given how the order alternatives behave in PEG + parsers (see :ref:`consequences of ordered choice section + ` for some background on this). In general, + try to define them in places where there is not a lot of alternatives. + +Error handling +~~~~~~~~~~~~~~ + +When a pegen-generated parser detects that an exception is raised, it will +**automatically stop parsing**, no matter what the current state of the parser +is and it will unwind the stack and report the exception. This means that if a +:ref:`rule action ` raises an exception all parsing will +stop at that exact point. This is done to allow to correctly propagate any +exception set by calling Python C-API functions. This also includes :exc:`SyntaxError` +exceptions and this is the main mechanism the parser uses to report custom syntax +error messages. + +.. note:: + Tokenizer errors are normally reported by raising exceptions but some special + tokenizer errors such as unclosed parenthesis will be reported only after the + parser finishes without returning anything. + +How Syntax errors are reported +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +As described previously in the :ref:`how PEG parsers work section +`, PEG parsers don't have a defined concept of where +errors happened in the grammar, because a rule failure doesn't imply a +parsing failure like in context free grammars. This means that some heuristic +has to be used to report generic errors unless something is explicitly declared +as an error in the grammar. + +To report generic syntax errors, pegen uses a common heuristic in PEG parsers: +the location of *generic* syntax errors is reported in the furthest token that +was attempted to be matched but failed. This is only done if parsing has failed +(the parser returns ``NULL`` in C or ``None`` in Python) but no exception has +been raised. + +.. caution:: + Positive and negative lookaheads will try to match a token so they will affect + the location of generic syntax errors. Use them carefully at boundaries + between rules. + +As the Python grammar was primordially written as an LL(1) grammar, this heuristic +has an extremely high success rate, but some PEG features can have small effects, +such as :ref:`positive lookaheads ` and +:ref:`negative lookaheads `. + +To generate more precise syntax errors, custom rules are used. This is a common practice +also in context free grammars: the parser will try to accept some construct that is known +to be incorrect just to report a specific syntax error for that construct. In pegen grammars, +these rules start with the ``invalid_`` prefix. This is because trying to match these rules +normally has a performance impact on parsing (and can also affect the 'correct' grammar itself +in some tricky cases, depending on the ordering of the rules) so the generated parser acts in +two phases: + +1. The first phase will try to parse the input stream without taking into account rules that + start with the ``invalid_`` prefix. If the parsing succeeds it will return the generated AST + and the second phase will not be attempted. + +2. If the first phase failed, a second parsing attempt is done including the rules that start + with an ``invalid_`` prefix. By design this attempt **cannot succeed** and is only executed + to give to the invalid rules a chance to detect specific situations where custom, more precise, + syntax errors can be raised. This also allows to trade a bit of performance for precision reporting + errors: given that we know that the input text is invalid, there is no need to be fast because + the interpreter is going to stop anyway. + +.. important:: + When defining invalid rules: + + * Make sure all custom invalid rules raise :exc:`SyntaxError` exceptions (or a subclass of it). + * Make sure **all** invalid rules start with the ``invalid_`` prefix to not + impact performance of parsing correct Python code. + * Make sure the parser doesn't behave differently for regular rules when you introduce invalid rules + (see the :ref:`how PEG parsers work section ` for more information). + +You can find a collection of macros to raise specialized syntax errors in the +:file:`Parser/pegen.h` header file. These macros allow also to report ranges for +the custom errors that will be highlighted in the tracebacks that will be +displayed when the error is reported. + +.. tip:: + A good way to test if an invalid rule will be triggered when you expect is to test if introducing + a syntax error **after** valid code triggers the rule or not. For example: :: + + $ 42 + + Should trigger the syntax error in the ``$`` character. If your rule is not correctly defined this + won't happen. For example, if you try to define a rule to match Python 2 style ``print`` statements + to make a better error message and you define it as: :: + + invalid_print: "print" expression + + This will **seem** to work because the parser will correctly parse ``print(something)`` because it is valid + code and the second phase will never execute but if you try to parse ``print(something) $ 3`` the first pass + of the parser will fail (because of the ``$``) and in the second phase, the rule will match the + ``print(something)`` as ``print`` followed by the variable ``something`` between parentheses and the error + will be reported there instead of the ``$`` character. + +Generating AST objects +~~~~~~~~~~~~~~~~~~~~~~ + +The output of the C parser used by CPython that is generated by the +:file:`Grammar/Python.gram` grammar file is a Python AST object (using C +structures). This means that the actions in the grammar file generate AST objects +when they succeed. Constructing these objects can be quite cumbersome (see +the :ref:`AST compiler section ` for more information +on how these objects are constructed and how they are used by the compiler) so +special helper functions are used. These functions are declared in the +:file:`Parser/pegen.h` header file and defined in the :file:`Parser/pegen.c` +file. These functions allow you to join AST sequences, get specific elements +from them or to do extra processing on the generated tree. + +.. caution:: + Actions must **never** be used to accept or reject rules. It may be tempting + in some situations to write a very generic rule and then check the generated + AST to decide if is valid or not but this will render the `official grammar + `_ partially incorrect + (because actions are not included) and will make it more difficult for other + Python implementations to adapt the grammar to their own needs. + +As a general rule, if an action spawns multiple lines or requires something more +complicated than a single expression of C code, is normally better to create a +custom helper in :file:`Parser/pegen.c` and expose it in the +:file:`Parser/pegen.h` header file so it can be used from the grammar. + +If the parsing succeeds, the parser **must** return a **valid** AST object. + +Testing +------- + +There are three files that contain tests for the grammar and the parser: + +* `Lib/test/test_grammar.py`. +* `Lib/test/test_syntax.py`. +* `Lib/test/test_exceptions.py`. + +Check the contents of these files to know which is the best place to place new tests depending +on the nature of the new feature you are adding. + +Tests for the parser generator itself can be found in the :file:`Lib/test/test_peg_generator` directory. + + +Debugging generated parsers +--------------------------- + +Making experiments +~~~~~~~~~~~~~~~~~~ + +As the generated C parser is the one used by Python, this means that if something goes wrong when adding some +new rules to the grammar you cannot correctly compile and execute Python anymore. This makes it a bit challenging +to debug when something goes wrong, especially when making experiments. + +For this reason it is a good idea to experiment first by generating a Python parser. To do this, you can go to the +:file:`Tools/peg_generator/` directory on the CPython repository and manually call the parser generator by executing: + +.. code-block:: shell + + $ python -m pegen python ~/github/pegen/data/expr.gram + +This will generate a file called :file:`parse.py` in the same directory that you can use to parse some input: + +.. code-block:: shell + + $ python parse.py file_with_source_code_to_test.py + +As the generated :file:`parse.py` file is just Python code, you can modify it and add breakpoints to debug or +better understand some complex situations. + + +Verbose mode +~~~~~~~~~~~~ + +When Python is compiled in debug mode (by adding ``--with-pydebug`` when running the configure step in Linux or by +adding ``-d`` when calling the :file:`PCbuild/python.bat` script in Windows), is possible to activate a **very** verbose +mode in the generated parser. This is very useful to debug the generated parser and to understand how it works, but it +can be a bit hard to understand at first. + +.. note:: + + When activating verbose mode in the Python parser, it is better to not use interactive mode as it can be much harder to + understand, because interactive mode involves some special steps compared to regular parsing. + +To activate verbose mode you can add the ``-d`` flag when executing Python: + +.. code-block:: shell + + $ python -d file_to_test.py + +This will print **a lot** of output to ``stderr`` so is probably better to dump it to a file for further analysis. The output +consists of trace lines with the following structure: + + ('>'|'-'|'+'|'!') []: ... + +Every line is indented by a different amount (````) depending on how deep the call stack is. The next +character marks the type of the trace: + +* ``>`` indicates that a rule is going to be attempted to be parsed. +* ``-`` indicates that a rule has failed to be parsed. +* ``+`` indicates that a rule has been parsed correctly. +* ``!`` indicates that an exception or an error has been detected and the parser is unwinding. + +The part indicates the current index in the token array, the + part indicates what rule is being parsed and the part +indicates what alternative within that rule is being attempted. + + +References +---------- + +.. [1] Ford, Bryan + http://pdos.csail.mit.edu/~baford/packrat/thesis + +.. [2] Medeiros et al. + https://arxiv.org/pdf/1207.0443.pdf + +.. [3] Warth et al. + http://web.cs.ucla.edu/~todd/research/pepm08.pdf diff --git a/requirements.txt b/requirements.txt index c90e30088..829318d20 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ Sphinx==4.1.2 -furo +furo<=2021.7.5b38 sphinx_copybutton>=0.3.3 From d9c1441301d494c2e751479fb45043b139733cc6 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Wed, 4 Aug 2021 21:54:27 +0100 Subject: [PATCH 056/793] Minor correction on the parser document --- parser.rst | 6 ------ 1 file changed, 6 deletions(-) diff --git a/parser.rst b/parser.rst index 71387b639..6364584b1 100644 --- a/parser.rst +++ b/parser.rst @@ -343,12 +343,6 @@ inside curly-braces, which specifies the return value of the alternative:: | first_alt1 first_alt2 { first_alt1 } | second_alt1 second_alt2 { second_alt1 } -If the action is omitted and C code is being generated, then there are two -different possibilities: - -1. If there’s a single name in the alternative, this gets returned. -2. If not, a dummy name object gets returned (this case should be avoided). - If the action is ommited, a default action is generated: * If there's a single name in the rule in the rule, it gets returned. From 14e6f3ea0c2731ddebf38224566d138b6ecd2d1f Mon Sep 17 00:00:00 2001 From: Pablo Galindo Date: Wed, 25 Aug 2021 11:52:23 +0100 Subject: [PATCH 057/793] Correct invocation command for testing the parser --- parser.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/parser.rst b/parser.rst index 6364584b1..27ce46d62 100644 --- a/parser.rst +++ b/parser.rst @@ -847,7 +847,7 @@ For this reason it is a good idea to experiment first by generating a Python par .. code-block:: shell - $ python -m pegen python ~/github/pegen/data/expr.gram + $ python -m pegen python This will generate a file called :file:`parse.py` in the same directory that you can use to parse some input: From 03d8f1bd581319a63aeedde5654485342a572e2c Mon Sep 17 00:00:00 2001 From: Brett Cannon Date: Thu, 26 Aug 2021 11:42:02 -0700 Subject: [PATCH 058/793] Add Ammar Askar and Ken Jin to the developer log --- developers.csv | 2 ++ 1 file changed, 2 insertions(+) diff --git a/developers.csv b/developers.csv index b37e5a15e..5b4551629 100644 --- a/developers.csv +++ b/developers.csv @@ -1,3 +1,5 @@ +Ken Jin,Fidget-Spinner,2021-08-26,, +Ammar Askar,ammaraskar,2021-07-30,, Irit Katriel,iritkatriel,2021-05-10,, Batuhan Taskaya,isidentical,2020-11-08,, Brandt Bucher,brandtbucher,2020-09-14,, From dd55ff4527b8211c01131ed11db28cabe450991d Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Fri, 27 Aug 2021 17:23:46 +0800 Subject: [PATCH 059/793] Describe test-with-buildbots label for triagers (#740) --- triaging.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/triaging.rst b/triaging.rst index 6c7468b18..7419aaa05 100644 --- a/triaging.rst +++ b/triaging.rst @@ -170,6 +170,12 @@ type-security type-tests Used for PRs that exclusively involve changes to the tests. +test-with-buildbots + Used on PRs to test the latest commit with the buildbot fleet. Generally for + PRs with large code changes requiring more testing before merging. This + may take multiple hours to complete. Triagers can also stop a stuck build + using the web interface. + Fields in the Issue Tracker --------------------------- From 1e0275e3b2477cefbf6aed9920b7d8e9e9b85654 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Wed, 8 Sep 2021 17:44:09 +0200 Subject: [PATCH 060/793] Mention DiR role in devcycle.rst --- devcycle.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/devcycle.rst b/devcycle.rst index e8bb60934..cfb18430a 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -321,6 +321,7 @@ Current Administrators | | Maintainer of buildbot.python.org | | +-------------------+----------------------------------------------------------+-----------------+ | Łukasz Langa | Python 3.8 and 3.9 Release Manager | ambv | +| | PSF CPython Developer in Residence 2021-2022 | | +-------------------+----------------------------------------------------------+-----------------+ | Ned Deily | Python 3.6 and 3.7 Release Manager | ned-deily | +-------------------+----------------------------------------------------------+-----------------+ From eeadb62751fc9011830a637e492debc18a214ce3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Wed, 8 Sep 2021 17:44:34 +0200 Subject: [PATCH 061/793] Update planned EOL for all feature branches, add 3.11 release PEP --- index.rst | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/index.rst b/index.rst index 37afb6e85..5f7d82632 100644 --- a/index.rst +++ b/index.rst @@ -97,11 +97,11 @@ Status of Python branches +------------------+--------------+-------------+----------------+----------------+-----------------------+ | Branch | Schedule | Status | First release | End-of-life | Release manager | +==================+==============+=============+================+================+=======================+ -| main | *TBD* | features | *TBD* | *TBD* | Pablo Galindo Salgado | +| main | :pep:`664` | features | *2022-10-03* | *2027-10* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ -| 3.10 | :pep:`619` | prerelease | *2021-10-04* | *TBD* | Pablo Galindo Salgado | +| 3.10 | :pep:`619` | prerelease | *2021-10-04* | *2026-10* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ -| 3.9 | :pep:`596` | bugfix | 2020-10-05 | *TBD* | Łukasz Langa | +| 3.9 | :pep:`596` | bugfix | 2020-10-05 | *2025-10* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.8 | :pep:`569` | security | 2019-10-14 | *2024-10* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ @@ -112,6 +112,8 @@ Status of Python branches .. Remember to update the end-of-life table in devcycle.rst. +Dates in *italic* are scheduled and can be adjusted. + The main branch is currently the future Python 3.11, and is the only branch that accepts new features. The latest release for each Python version can be found on the `download page `_. @@ -127,13 +129,12 @@ Status: but new source-only versions can be released :end-of-life: release cycle is frozen; no further changes can be pushed to it. -Dates in *italic* are scheduled and can be adjusted. +See also the :ref:`devcycle` page for more information about branches. By default, the end-of-life is scheduled 5 years after the first release, but can be adjusted by the release manager of each branch. All Python 2 versions have reached end-of-life. -See also the :ref:`devcycle` page for more information about branches. .. _contributing: From f5c76b915e04bdbe246dfece7033ff02e1a1304c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C5=81ukasz=20Langa?= Date: Wed, 8 Sep 2021 17:51:07 +0200 Subject: [PATCH 062/793] Add missing comma --- devcycle.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/devcycle.rst b/devcycle.rst index cfb18430a..35bd96ab9 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -320,7 +320,7 @@ Current Administrators | Pablo Galindo | Python 3.10 and 3.11 Release Manager, | pablogsal | | | Maintainer of buildbot.python.org | | +-------------------+----------------------------------------------------------+-----------------+ -| Łukasz Langa | Python 3.8 and 3.9 Release Manager | ambv | +| Łukasz Langa | Python 3.8 and 3.9 Release Manager, | ambv | | | PSF CPython Developer in Residence 2021-2022 | | +-------------------+----------------------------------------------------------+-----------------+ | Ned Deily | Python 3.6 and 3.7 Release Manager | ned-deily | From 54cd7c1dcd9ddc082ded55651696052fc2700c59 Mon Sep 17 00:00:00 2001 From: Ken Jin <28750310+Fidget-Spinner@users.noreply.github.com> Date: Fri, 10 Sep 2021 23:41:33 +0800 Subject: [PATCH 063/793] Link to configure docs in compilation guide (#743) --- setup.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/setup.rst b/setup.rst index 24b7c97d2..0c69af695 100644 --- a/setup.rst +++ b/setup.rst @@ -128,6 +128,8 @@ when you shouldn't is if you are taking performance measurements). Even when working only on pure Python code the pydebug build provides several useful checks that one should not skip. +.. seealso:: The effects of various configure and build flags are documented in + the `Python configure docs `_. .. _unix-compiling: From 7a7a663e7f0d1ad6360bb35b58dd2357d3b7bf17 Mon Sep 17 00:00:00 2001 From: Dmitriy Fishman Date: Mon, 27 Sep 2021 14:55:18 +0300 Subject: [PATCH 064/793] Typo fix in committing.rst (GH-749) --- committing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/committing.rst b/committing.rst index 213a2292e..9f1951ba7 100644 --- a/committing.rst +++ b/committing.rst @@ -102,7 +102,7 @@ These are the two exceptions: change and the original** ``NEWS`` **entry remains valid**, then no additional entry is needed. -If a change needs an entry in ``What's New in Python``, then it very +If a change needs an entry in ``What's New in Python``, then it is very likely not suitable for including in a maintenance release. ``NEWS`` entries go into the ``Misc/NEWS.d`` directory as individual files. The From b1efd6a9ee74fad66d853ad13856f815a2822f8f Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Mon, 4 Oct 2021 01:05:39 +0100 Subject: [PATCH 065/793] Add instructions on running autoreconf with pkg-config (#750) --- setup.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/setup.rst b/setup.rst index 0c69af695..42cbe1758 100644 --- a/setup.rst +++ b/setup.rst @@ -447,7 +447,14 @@ example, ``autoconf`` by itself will not regenerate ``pyconfig.h.in``. appropriate. Python's ``configure.ac`` script typically requires a specific version of -Autoconf. At the moment, this reads: ``AC_PREREQ(2.69)``. +Autoconf. At the moment, this reads: ``AC_PREREQ(2.69)``. It also requires +to have the ``autoconf-archive`` and ``pkg-config`` utilities installed in +the system and the ``pkg.m4`` macro file located in the appropriate ``alocal`` +location. You can easily check if this is correctly configured by running: + +.. code-block:: bash + + ls $(aclocal --print-ac-dir) | grep pkg.m4 If the system copy of Autoconf does not match this version, you will need to install your own copy of Autoconf. From ebc5777ac4332f4507f73331057f670715e44f47 Mon Sep 17 00:00:00 2001 From: Ned Deily Date: Mon, 4 Oct 2021 16:09:38 -0400 Subject: [PATCH 066/793] update for 3.10 release --- index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.rst b/index.rst index 5f7d82632..3e311ae4a 100644 --- a/index.rst +++ b/index.rst @@ -99,7 +99,7 @@ Status of Python branches +==================+==============+=============+================+================+=======================+ | main | :pep:`664` | features | *2022-10-03* | *2027-10* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ -| 3.10 | :pep:`619` | prerelease | *2021-10-04* | *2026-10* | Pablo Galindo Salgado | +| 3.10 | :pep:`619` | bugfix | 2021-10-04 | *2026-10* | Pablo Galindo Salgado | +------------------+--------------+-------------+----------------+----------------+-----------------------+ | 3.9 | :pep:`596` | bugfix | 2020-10-05 | *2025-10* | Łukasz Langa | +------------------+--------------+-------------+----------------+----------------+-----------------------+ From 0dc9c92bafb58e61cde43bf57b3f31b65d78fd46 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Tue, 19 Oct 2021 09:20:33 -0400 Subject: [PATCH 067/793] update Python org owners (#753) --- devcycle.rst | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/devcycle.rst b/devcycle.rst index 35bd96ab9..e6e20b4fe 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -260,7 +260,7 @@ This role is paramount to the security of the Python Language, Community, and Infrastructure. The Executive Director of the Python Software Foundation delegates authority on -GitHub Organization Owner Status to Ee W. Durbin III - Python Software +GitHub Organization Owner Status to Ee Durbin - Python Software Foundation Director of Infrastructure. Common reasons for this role are: Infrastructure Staff Membership, Python Software Foundation General Counsel, and Python Software Foundation Staff as fallback. @@ -285,10 +285,14 @@ Current Owners +----------------------+--------------------------------+-----------------+ | Ewa Jodlowska | PSF Executive Director | ejodlowska | +----------------------+--------------------------------+-----------------+ -| Ee W. Durbin III | PSF Director of Infrastructure | ewdurbin | +| Ee Durbin | PSF Director of Infrastructure | ewdurbin | +----------------------+--------------------------------+-----------------+ | Van Lindberg | PSF General Counsel | VanL | +----------------------+--------------------------------+-----------------+ +| Ezio Melotti | roundup -> github migration | ezio-melotti | ++----------------------+--------------------------------+-----------------+ +| Łukasz Langa | CPython Developr in Residence | ambv | ++----------------------+--------------------------------+-----------------+ Repository Administrator Role Policy ------------------------------------ From f34e3442c3e80683df4c738352e8139992fb08b0 Mon Sep 17 00:00:00 2001 From: Zachary Ware Date: Tue, 19 Oct 2021 11:08:03 -0500 Subject: [PATCH 068/793] Typo fix --- devcycle.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/devcycle.rst b/devcycle.rst index e6e20b4fe..5c1b4b53e 100644 --- a/devcycle.rst +++ b/devcycle.rst @@ -291,7 +291,7 @@ Current Owners +----------------------+--------------------------------+-----------------+ | Ezio Melotti | roundup -> github migration | ezio-melotti | +----------------------+--------------------------------+-----------------+ -| Łukasz Langa | CPython Developr in Residence | ambv | +| Łukasz Langa | CPython Developer in Residence | ambv | +----------------------+--------------------------------+-----------------+ Repository Administrator Role Policy From c4f3c7300c59b19a91cc66f3363b8f5985635f12 Mon Sep 17 00:00:00 2001 From: Wey-Han Liaw Date: Sun, 24 Oct 2021 02:44:19 -0700 Subject: [PATCH 069/793] Change the coordinator of the zh-tw translation (#752) --- documenting.rst | 7 ++++--- experts.rst | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/documenting.rst b/documenting.rst index a0521ce9e..6b91652ca 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1641,9 +1641,9 @@ in production, other are work in progress: +-----------------+-------------------------------+----------------------------+ | Spanish (es) | Raúl Cumplido | `GitHub `_ | +-----------------+-------------------------------+----------------------------+ -| Traditional | 廖偉涵 Adrian Liaw | `GitHub `_ | -| Chinese | | `Transifex `_ | -| (zh-tw) | | `Doc `_ | +| Traditional | `王威翔 Matt Wang | `GitHub `_ | +| Chinese | `_, | `Transifex `_ | +| (zh-tw) | Josix Wang | `Doc `_ | +-----------------+-------------------------------+----------------------------+ | Turkish (tr) | | `GitHub `_ | +-----------------+-------------------------------+----------------------------+ @@ -1655,6 +1655,7 @@ in production, other are work in progress: .. _bpo_mdk: https://bugs.python.org/user23063 .. _bpo_oonid: https://bugs.python.org/user32660 .. _bpo_zhsj: https://bugs.python.org/user24811 +.. _bpo_mattwang44: https://bugs.python.org/user39654 .. _chat_pt_br: https://t.me/pybr_i18n .. _doc_ja: https://docs.python.org/ja/ .. _doc_ko: https://docs.python.org/ko/ diff --git a/experts.rst b/experts.rst index dc84bdfb9..f1495ca6c 100644 --- a/experts.rst +++ b/experts.rst @@ -368,6 +368,6 @@ Korean flowdas Bengali India kushal.das Hungarian gbtami Portuguese rougeth -Chinese (TW) adrianliaw +Chinese (TW) mattwang44, josix Chinese (CN) zhsj ============= ============ From 2e6b3279711d4d528ee75533c46acb87519d62c0 Mon Sep 17 00:00:00 2001 From: Christian Heimes Date: Thu, 28 Oct 2021 12:24:45 +0300 Subject: [PATCH 070/793] Update Debian build requirements (GH-755) --- setup.rst | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/setup.rst b/setup.rst index 42cbe1758..782a498a5 100644 --- a/setup.rst +++ b/setup.rst @@ -328,13 +328,15 @@ Then you should update the packages index:: Now you can install the build dependencies via ``apt``:: $ sudo apt-get build-dep python3 + $ sudo apt-get install pkg-config If you want to build all optional modules, install the following packages and their dependencies:: - $ sudo apt-get install build-essential gdb lcov libbz2-dev libffi-dev \ - libgdbm-dev liblzma-dev libncurses5-dev libreadline6-dev \ - libsqlite3-dev libssl-dev lzma lzma-dev tk-dev uuid-dev zlib1g-dev + $ sudo apt-get install build-essential gdb lcov pkg-config \ + libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \ + libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \ + lzma lzma-dev tk-dev uuid-dev zlib1g-dev .. _MacOS: From 8d13a975939b9d9b93f1cff28b0ade3be7744695 Mon Sep 17 00:00:00 2001 From: Itamar Ostricher Date: Tue, 2 Nov 2021 22:14:35 -0700 Subject: [PATCH 071/793] Minor typo fix in Issue Tracking page (#758) --- tracker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tracker.rst b/tracker.rst index 9f28aa3c6..367306dee 100644 --- a/tracker.rst +++ b/tracker.rst @@ -30,7 +30,7 @@ already been reported. Checking if the problem is an existing issue will: * determine if additional information, such as how to replicate the issue, is needed -To do see if the issue already exists, search the bug database using the +To see if an issue already exists, search the bug database using the search box on the top of the issue tracker page. An `advanced search`_ is also available by clicking on "Search" in the sidebar. From 5db50fc080e102b194a8e7ca46bc6f94cff3bd0d Mon Sep 17 00:00:00 2001 From: Abdur-Rahmaan Janhangeer Date: Mon, 8 Nov 2021 00:08:40 +0400 Subject: [PATCH 072/793] Update Arabic coordinator (#760) --- documenting.rst | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/documenting.rst b/documenting.rst index 6b91652ca..4649bab3d 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1595,7 +1595,8 @@ in production, other are work in progress: +-----------------+-------------------------------+----------------------------+ | Language | Contact | Links | +=================+===============================+============================+ -| Arabic (ar) | Ibrahim Elbouhissi | `GitHub `_ | +| Arabic (ar) | `Abdur-Rahmaan Janhangeer | `GitHub `_ | +| | `_ | | +-----------------+-------------------------------+----------------------------+ | Bengali as | `Kushal Das `_ | `GitHub `_ | | spoken in | | | @@ -1654,6 +1655,7 @@ in production, other are work in progress: .. _bpo_kushal: https://bugs.python.org/user16382 .. _bpo_mdk: https://bugs.python.org/user23063 .. _bpo_oonid: https://bugs.python.org/user32660 +.. _bpo_osdotsystem: https://bugs.python.org/user28057 .. _bpo_zhsj: https://bugs.python.org/user24811 .. _bpo_mattwang44: https://bugs.python.org/user39654 .. _chat_pt_br: https://t.me/pybr_i18n From 75e0e7ece4e967cac88d4a707a5c52c98d9b33c6 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sun, 7 Nov 2021 23:21:36 +0000 Subject: [PATCH 073/793] Fix grammar in the title of the Parser guide (#762) --- parser.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/parser.rst b/parser.rst index 27ce46d62..491074022 100644 --- a/parser.rst +++ b/parser.rst @@ -1,6 +1,6 @@ .. _parser: -Guide of CPython's Parser +Guide to CPython's Parser ========================= :Author: Pablo Galindo Salgado From 46590a51d1e1224e9b987cb4c33b313937fd5ecd Mon Sep 17 00:00:00 2001 From: Itamar Ostricher Date: Sun, 14 Nov 2021 10:29:40 -0800 Subject: [PATCH 074/793] Fix typo in garbage_collector (#768) Minor fix / nit --- garbage_collector.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/garbage_collector.rst b/garbage_collector.rst index 0dbac59fc..7e15a4ea2 100644 --- a/garbage_collector.rst +++ b/garbage_collector.rst @@ -303,7 +303,7 @@ In order to limit the time each garbage collection takes, the GC uses a popular optimization: generations. The main idea behind this concept is the assumption that most objects have a very short lifespan and can thus be collected shortly after their creation. This has proven to be very close to the reality of many Python programs as -many temporarily objects are created and destroyed very fast. The older an object is +many temporary objects are created and destroyed very fast. The older an object is the less likely it is that it will become unreachable. To take advantage of this fact, all container objects are segregated into From 3036184234b4821b380bec0dd4e2bbf81ae812b1 Mon Sep 17 00:00:00 2001 From: Itamar Ostricher Date: Sun, 14 Nov 2021 10:29:58 -0800 Subject: [PATCH 075/793] Update reference to the parser from LL(1) to PEG (#767) --- langchanges.rst | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/langchanges.rst b/langchanges.rst index f7984d608..3dbaf6d47 100644 --- a/langchanges.rst +++ b/langchanges.rst @@ -46,9 +46,8 @@ the change. This process is the Python Enhancement Proposal (PEP) process. You will first need a PEP that you will present to python-ideas. You may be a little hazy on the technical details as various core developers can help with that, but do realize that if you do not present your idea to python-ideas or -python-list ahead of time you may find out it is technically not possible -(e.g., Python's parser will not support the grammar change as it is an LL(1) -parser). Expect extensive comments on the PEP, some of which will be negative. +python-list ahead of time you may find out it is technically not possible. +Expect extensive comments on the PEP, some of which will be negative. Once your PEP has been modified to be of proper quality and to take into account comments made on python-ideas, it may proceed to python-dev. There it From 90bd4308657ea61ef740c648ac87e49dde819fa3 Mon Sep 17 00:00:00 2001 From: Dino Viehland Date: Mon, 15 Nov 2021 13:29:39 -0800 Subject: [PATCH 076/793] Update motivations.rst for Dino Way out of date... --- motivations.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/motivations.rst b/motivations.rst index f1a1a41cc..27361311d 100644 --- a/motivations.rst +++ b/motivations.rst @@ -244,16 +244,16 @@ participating in the CPython core development process: .. topic:: Dino Viehland (United States) - * Microsoft: ``_ (Software Engineer) - * Email address: dinov@microsoft.com + * Meta (Software Engineer) + * Email address: dinoviehland@gmail.com Dino started working with Python in 2005 by working on IronPython, an implementation of Python running on .NET. He was one of the primary developers on the project for 6 years. After that he started the Python Tools for Visual Studio project focusing on providing advanced code completion and debugging features for Python. Today he works on - `Azure Notebooks `_ bringing the Python based - Jupyter notebook as a hosted on-line service. + `Cinder `_ improving Python + performance for Instagram. .. topic:: Carol Willing (United States) From 3f372f9a1c5416df0f3efaa529c23c0a2fc4aac3 Mon Sep 17 00:00:00 2001 From: Arthur Milchior Date: Tue, 16 Nov 2021 16:06:47 +0100 Subject: [PATCH 077/793] Replacing "country" by "translation" (GH-757) While it is easy to understand what was meant by "country", this is inaccurate. Some countries have multiple languages, each having their translation (e.g. Canada with English and French) and some languages are spoken in many countries, and the regional variation are small enough that there will probably never be multiple translations. (While Canadian French and France French are so different that it's sometimes hard for French native to understand French Canadian speaking, I doubt anybody will want to create a ca-fr translation, given that the technical language will probably be almost the same.) --- documenting.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documenting.rst b/documenting.rst index 4649bab3d..852b16237 100644 --- a/documenting.rst +++ b/documenting.rst @@ -1771,7 +1771,7 @@ Here's what's we're using: How a coordinator is elected? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -There is no election, each country have to sort this out. Here are some suggestions. +There is no election, each translation have to sort this out. Here are some suggestions. - Coordinator requests are to be public on doc-sig mailing list. - If the given language have a native core dev, the core dev have its From 266f1c0eb60cc2165f833da0d6d1d6f4772ea2fd Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 16 Nov 2021 09:56:22 -0800 Subject: [PATCH 078/793] Update furo requirement from <=2021.7.5b38 to <2021.11.16 (#770) Updates the requirements on [furo](https://github.com/pradyunsg/furo) to permit the latest version. - [Release notes](https://github.com/pradyunsg/furo/releases) - [Changelog](https://github.com/pradyunsg/furo/blob/main/docs/changelog.md) - [Commits](https://github.com/pradyunsg/furo/compare/2020.08.14.beta5...2021.11.15) --- updated-dependencies: - dependency-name: furo dependency-type: direct:production ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/requirements.txt b/requirements.txt index 829318d20..bd66e003c 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ Sphinx==4.1.2 -furo<=2021.7.5b38 +furo<2021.11.16 sphinx_copybutton>=0.3.3 From 1ff3dc740e946aace6b62ff85821eafd4b98d0a4 Mon Sep 17 00:00:00 2001 From: Itamar Ostricher Date: Fri, 19 Nov 2021 00:10:26 -0800 Subject: [PATCH 079/793] Four fixes in parser page (#772) Two duplications, a typo, and minor grammar fix. --- parser.rst | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/parser.rst b/parser.rst index 491074022..06434fc57 100644 --- a/parser.rst +++ b/parser.rst @@ -110,7 +110,7 @@ the following two rules (in these examples, a token is an individual character): In a regular EBNF grammar, both rules specify the language ``{aa, aaa}`` but in PEG, one of these two rules accepts the string ``aaa`` but not the string -``aa``. The other does the opposite -- it accepts the string the string ``aa`` +``aa``. The other does the opposite -- it accepts the string ``aa`` but not the string ``aaa``. The rule ``('a'|'aa')'a'`` does not accept ``aaa`` because ``'a'|'aa'`` consumes the first ``a``, letting the final ``a`` in the rule consume the second, and leaving out the third ``a``. @@ -345,14 +345,14 @@ inside curly-braces, which specifies the return value of the alternative:: If the action is ommited, a default action is generated: -* If there's a single name in the rule in the rule, it gets returned. +* If there's a single name in the rule, it gets returned. * If there is more than one name in the rule, a collection with all parsed expressions gets returned (the type of the collection will be different in C and Python). This default behaviour is primarily made for very simple situations and for -debugging pourposes. +debugging purposes. The full meta-grammar for the grammars supported by the PEG generator is: @@ -863,7 +863,7 @@ Verbose mode ~~~~~~~~~~~~ When Python is compiled in debug mode (by adding ``--with-pydebug`` when running the configure step in Linux or by -adding ``-d`` when calling the :file:`PCbuild/python.bat` script in Windows), is possible to activate a **very** verbose +adding ``-d`` when calling the :file:`PCbuild/python.bat` script in Windows), it is possible to activate a **very** verbose mode in the generated parser. This is very useful to debug the generated parser and to understand how it works, but it can be a bit hard to understand at first. From 9d880b6f54aa2ee193fccdb1242a52923df8c8aa Mon Sep 17 00:00:00 2001 From: Carl Friedrich Bolz-Tereick Date: Fri, 19 Nov 2021 22:16:43 +0100 Subject: [PATCH 080/793] =?UTF-8?q?Improvements=20to=20"Guide=20to=20CPyth?= =?UTF-8?q?on=E2=80=99s=20Parser"=20(#763)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * put references to names from rules into ``backticks`` * two typos * mention that ~ is called "the cut" (The grammar gives that name, and I as a prolog programmer was looking for the terminology ;-)) --- parser.rst | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/parser.rst b/parser.rst index 06434fc57..6001051a6 100644 --- a/parser.rst +++ b/parser.rst @@ -169,7 +169,7 @@ Python-style comments. ``e1 e2`` ''''''''' -Match e1, then match e2. +Match ``e1``, then match ``e2``. :: @@ -178,7 +178,7 @@ Match e1, then match e2. ``e1 | e2`` ''''''''''' -Match e1 or e2. +Match ``e1`` or ``e2``. The first alternative can also appear on the line after the rule name for formatting purposes. In that case, a \| must be used before the @@ -193,7 +193,7 @@ first alternative, like so: ``( e )`` ''''''''' -Match e. +Match ``e``. :: @@ -209,7 +209,7 @@ operator together with the repeat operators: ``[ e ] or e?`` ''''''''''''''' -Optionally match e. +Optionally match ``e``. :: @@ -225,7 +225,7 @@ optional: ``e*`` '''''' -Match zero or more occurrences of e. +Match zero or more occurrences of ``e``. :: @@ -234,7 +234,7 @@ Match zero or more occurrences of e. ``e+`` '''''' -Match one or more occurrences of e. +Match one or more occurrences of ``e``. :: @@ -243,7 +243,7 @@ Match one or more occurrences of e. ``s.e+`` '''''''' -Match one or more occurrences of e, separated by s. The generated parse +Match one or more occurrences of ``e``, separated by ``s``. The generated parse tree does not include the separator. This is otherwise identical to ``(e (s e)*)``. @@ -256,14 +256,14 @@ tree does not include the separator. This is otherwise identical to .. _peg-positive-lookahead: -Succeed if e can be parsed, without consuming any input. +Succeed if ``e`` can be parsed, without consuming any input. ``!e`` '''''' .. _peg-negative-lookahead: -Fail if e can be parsed, without consuming any input. +Fail if ``e`` can be parsed, without consuming any input. An example taken from the Python grammar specifies that a primary consists of an atom, which is not followed by a ``.`` or a ``(`` or a @@ -276,14 +276,15 @@ consists of an atom, which is not followed by a ``.`` or a ``(`` or a ``~`` '''''' -Commit to the current alternative, even if it fails to parse. +Commit to the current alternative, even if it fails to parse (this is called +the "cut"). :: rule_name: '(' ~ some_rule ')' | some_alt In this example, if a left parenthesis is parsed, then the other -alternative won’t be considered, even if some_rule or ‘)’ fail to be +alternative won’t be considered, even if some_rule or ``)`` fail to be parsed. Left recursion @@ -343,7 +344,7 @@ inside curly-braces, which specifies the return value of the alternative:: | first_alt1 first_alt2 { first_alt1 } | second_alt1 second_alt2 { second_alt1 } -If the action is ommited, a default action is generated: +If the action is omitted, a default action is generated: * If there's a single name in the rule, it gets returned. From ed2af4311a8db23b0d6267fea45090c22a3d8ce7 Mon Sep 17 00:00:00 2001 From: Carl Friedrich Bolz-Tereick Date: Fri, 19 Nov 2021 22:16:59 +0100 Subject: [PATCH 081/793] add warning about actions mutating things (#766) --- parser.rst | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/parser.rst b/parser.rst index 6001051a6..c773b00c7 100644 --- a/parser.rst +++ b/parser.rst @@ -355,6 +355,15 @@ If the action is omitted, a default action is generated: This default behaviour is primarily made for very simple situations and for debugging purposes. +.. warning:: + + It's important that the actions don't mutate any AST nodes that are passed + into them via variables referring to other rules. The reason for mutation + being not allowed is that the AST nodes are cached by memoization and could + potentially be reused in a different context, where the mutation would be + invalid. If an action needs to change an AST node, it should instead make a + new copy of the node and change that. + The full meta-grammar for the grammars supported by the PEG generator is: :: From f3ecfb8a525a971662413c1ecdc0c8d8fc540393 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Sun, 21 Nov 2021 20:41:08 +0000 Subject: [PATCH 082/793] Update AST actions helper location in the PEG parser (#773) --- parser.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/parser.rst b/parser.rst index c773b00c7..1bc795dec 100644 --- a/parser.rst +++ b/parser.rst @@ -808,7 +808,7 @@ when they succeed. Constructing these objects can be quite cumbersome (see the :ref:`AST compiler section ` for more information on how these objects are constructed and how they are used by the compiler) so special helper functions are used. These functions are declared in the -:file:`Parser/pegen.h` header file and defined in the :file:`Parser/pegen.c` +:file:`Parser/pegen.h` header file and defined in the :file:`Parser/action_helpers.c` file. These functions allow you to join AST sequences, get specific elements from them or to do extra processing on the generated tree. @@ -822,7 +822,7 @@ from them or to do extra processing on the generated tree. As a general rule, if an action spawns multiple lines or requires something more complicated than a single expression of C code, is normally better to create a -custom helper in :file:`Parser/pegen.c` and expose it in the +custom helper in :file:`Parser/action_helpers.c` and expose it in the :file:`Parser/pegen.h` header file so it can be used from the grammar. If the parsing succeeds, the parser **must** return a **valid** AST object. From 0b95382d21c77eed3f496585844d683f43603aa9 Mon Sep 17 00:00:00 2001 From: Ee Durbin Date: Thu, 2 Dec 2021 13:40:16 -0500 Subject: [PATCH 083/793] rename default branch to main (#776) also covers a handful of external references who have similarly made this change --- .github/workflows/ci.yml | 2 +- runtests.rst | 2 +- setup.rst | 2 +- tools/templates/customsourcelink.html | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 09cd927e7..a01ee9f37 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -3,7 +3,7 @@ name: Tests on: pull_request: push: - branches: master + branches: main jobs: test: diff --git a/runtests.rst b/runtests.rst index 2d6c3eb48..e018faa25 100644 --- a/runtests.rst +++ b/runtests.rst @@ -132,4 +132,4 @@ Benchmarking is useful to test that a change does not degrade performance. `The Python Benchmark Suite `_ has a collection of benchmarks for all Python implementations. Documentation about running the benchmarks is in the `README.txt -`_ of the repo. +`_ of the repo. diff --git a/setup.rst b/setup.rst index 782a498a5..7ef2bc460 100644 --- a/setup.rst +++ b/setup.rst @@ -13,7 +13,7 @@ directory structure of the CPython source code. Alternatively, if you have `Docker `_ installed you might want to use `our official images -`_. These +`_. These contain the latest releases of several Python versions, along with git head, and are provided for development and testing purposes only. diff --git a/tools/templates/customsourcelink.html b/tools/templates/customsourcelink.html index 2487a0b04..a50734f18 100644 --- a/tools/templates/customsourcelink.html +++ b/tools/templates/customsourcelink.html @@ -3,7 +3,7 @@

{{ _('This Page') }}