Add an "optimize" parameter to compile() to control the optimization level, and provide an interface to it in py_compile, compileall and PyZipFile.

Author: birkenfeld

Doc/c-api/veryhigh.rst

@@ -230,6 +230,12 @@ the same library that the Python runtime is using.
 
 .. c:function:: PyObject* Py_CompileStringFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags)
 
+   This is a simplified interface to :c:func:`Py_CompileStringExFlags` below, with
+   *optimize* set to ``-1``.
+
+
+.. c:function:: PyObject* Py_CompileStringExFlags(const char *str, const char *filename, int start, PyCompilerFlags *flags, int optimize)
+
    Parse and compile the Python source code in *str*, returning the resulting code
    object.  The start token is given by *start*; this can be used to constrain the
    code which can be compiled and should be :const:`Py_eval_input`,
@@ -238,6 +244,14 @@ the same library that the Python runtime is using.
    :exc:`SyntaxError` exception messages.  This returns *NULL* if the code cannot
    be parsed or compiled.
 
+   The integer *optimize* specifies the optimization level of the compiler; a
+   value of ``-1`` selects the optimization level of the interpreter as given by
+   :option:`-O` options.  Explicit levels are ``0`` (no optimization;
+   ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false)
+   or ``2`` (docstrings are removed too).
+
+   .. versionadded:: 3.2
+
 
 .. c:function:: PyObject* PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals)
 

Doc/library/compileall.rst

@@ -58,7 +58,7 @@ compile Python sources.
 Public functions
 ----------------
 
-.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False)
+.. function:: compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None, quiet=False, legacy=False, optimize=-1)
 
    Recursively descend the directory tree named by *dir*, compiling all :file:`.py`
    files along the way.  The *maxlevels* parameter is used to limit the depth of
@@ -76,14 +76,23 @@ Public functions
    If *legacy* is true, old-style ``.pyc`` file path names are written,
    otherwise (the default), :pep:`3147`-style path names are written.
 
+   *optimize* specifies the optimization level for the compiler.  It is passed to
+   the built-in :func:`compile` function.
 
-.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False)
+   .. versionchanged:: 3.2
+      Added the *optimize* parameter.
+
+
+.. function:: compile_path(skip_curdir=True, maxlevels=0, force=False, legacy=False, optimize=-1)
 
    Byte-compile all the :file:`.py` files found along ``sys.path``. If
    *skip_curdir* is true (the default), the current directory is not included in
-   the search.  The *maxlevels* parameter defaults to ``0``, and the *force*
-   and *legacy* parameters default to ``False``. All are
-   passed to the :func:`compile_dir` function.
+   the search.  All other parameters are passed to the :func:`compile_dir`
+   function.
+
+   .. versionchanged:: 3.2
+      Added the *optimize* parameter.
+
 
 To force a recompile of all the :file:`.py` files in the :file:`Lib/`
 subdirectory and all its subdirectories::

Doc/library/functions.rst

@@ -174,7 +174,7 @@ are always available.  They are listed here in alphabetical order.
    type hierarchy in :ref:`types`.
 
 
-.. function:: compile(source, filename, mode, flags=0, dont_inherit=False)
+.. function:: compile(source, filename, mode, flags=0, dont_inherit=False, optimize=-1)
 
    Compile the *source* into a code or AST object.  Code objects can be executed
    by :func:`exec` or :func:`eval`.  *source* can either be a string or an AST
@@ -206,6 +206,12 @@ are always available.  They are listed here in alphabetical order.
    can be found as the :attr:`compiler_flag` attribute on the :class:`_Feature`
    instance in the :mod:`__future__` module.
 
+   The argument *optimize* specifies the optimization level of the compiler; the
+   default value of ``-1`` selects the optimization level of the interpreter as
+   given by :option:`-O` options.  Explicit levels are ``0`` (no optimization;
+   ``__debug__`` is true), ``1`` (asserts are removed, ``__debug__`` is false)
+   or ``2`` (docstrings are removed too).
+
    This function raises :exc:`SyntaxError` if the compiled source is invalid,
    and :exc:`TypeError` if the source contains null bytes.
 
@@ -218,7 +224,7 @@ are always available.  They are listed here in alphabetical order.
 
    .. versionchanged:: 3.2
       Allowed use of Windows and Mac newlines.  Also input in ``'exec'`` mode
-      does not have to end in a newline anymore.
+      does not have to end in a newline anymore.  Added the *optimize* parameter.
 
 
 .. function:: complex([real[, imag]])

Doc/library/py_compile.rst

@@ -22,7 +22,7 @@ byte-code cache files in the directory containing the source code.
    Exception raised when an error occurs while attempting to compile the file.
 
 
-.. function:: compile(file, cfile=None, dfile=None, doraise=False)
+.. function:: compile(file, cfile=None, dfile=None, doraise=False, optimize=-1)
 
    Compile a source file to byte-code and write out the byte-code cache  file.  The
    source code is loaded from the file name *file*.  The  byte-code is written to
@@ -37,6 +37,13 @@ byte-code cache files in the directory containing the source code.
    returns the path to byte-compiled file, i.e. whatever *cfile* value was
    used.
 
+   *optimize* controls the optimization level and is passed to the built-in
+   :func:`compile` function.  The default of ``-1`` selects the optimization
+   level of the current interpreter.
+
+   .. versionchanged:: 3.2
+      Added the *optimize* parameter.
+
 
 .. function:: main(args=None)
 

Doc/library/zipfile.rst

@@ -51,6 +51,7 @@ The module defines the following items:
 
 
 .. class:: PyZipFile
+   :noindex:
 
    Class for creating ZIP archives containing Python libraries.
 
@@ -318,37 +319,53 @@ The following data attributes are also available:
    string no longer than 65535 bytes.  Comments longer than this will be
    truncated in the written archive when :meth:`ZipFile.close` is called.
 
+
 .. _pyzipfile-objects:
 
 PyZipFile Objects
 -----------------
 
 The :class:`PyZipFile` constructor takes the same parameters as the
-:class:`ZipFile` constructor.  Instances have one method in addition to those of
-:class:`ZipFile` objects.
-
-
-.. method:: PyZipFile.writepy(pathname, basename='')
-
-   Search for files :file:`\*.py` and add the corresponding file to the archive.
-   The corresponding file is a :file:`\*.pyo` file if available, else a
-   :file:`\*.pyc` file, compiling if necessary.  If the pathname is a file, the
-   filename must end with :file:`.py`, and just the (corresponding
-   :file:`\*.py[co]`) file is added at the top level (no path information).  If the
-   pathname is a file that does not end with :file:`.py`, a :exc:`RuntimeError`
-   will be raised.  If it is a directory, and the directory is not a package
-   directory, then all the files :file:`\*.py[co]` are added at the top level.  If
-   the directory is a package directory, then all :file:`\*.py[co]` are added under
-   the package name as a file path, and if any subdirectories are package
-   directories, all of these are added recursively.  *basename* is intended for
-   internal use only.  The :meth:`writepy` method makes archives with file names
-   like this::
-
-      string.pyc                                # Top level name
-      test/__init__.pyc                         # Package directory
-      test/testall.pyc                          # Module test.testall
-      test/bogus/__init__.pyc                   # Subpackage directory
-      test/bogus/myfile.pyc                     # Submodule test.bogus.myfile
+:class:`ZipFile` constructor, and one additional parameter, *optimize*.
+
+.. class:: PyZipFile(file, mode='r', compression=ZIP_STORED, allowZip64=False, \
+                     optimize=-1)
+
+   .. versionadded:: 3.2
+      The *optimize* parameter.
+
+   Instances have one method in addition to those of :class:`ZipFile` objects:
+
+   .. method:: PyZipFile.writepy(pathname, basename='')
+
+      Search for files :file:`\*.py` and add the corresponding file to the
+      archive.
+
+      If the *optimize* parameter to :class:`PyZipFile` was not given or ``-1``,
+      the corresponding file is a :file:`\*.pyo` file if available, else a
+      :file:`\*.pyc` file, compiling if necessary.
+
+      If the *optimize* parameter to :class:`PyZipFile` was ``0``, ``1`` or
+      ``2``, only files with that optimization level (see :func:`compile`) are
+      added to the archive, compiling if necessary.
+
+      If the pathname is a file, the filename must end with :file:`.py`, and
+      just the (corresponding :file:`\*.py[co]`) file is added at the top level
+      (no path information).  If the pathname is a file that does not end with
+      :file:`.py`, a :exc:`RuntimeError` will be raised.  If it is a directory,
+      and the directory is not a package directory, then all the files
+      :file:`\*.py[co]` are added at the top level.  If the directory is a
+      package directory, then all :file:`\*.py[co]` are added under the package
+      name as a file path, and if any subdirectories are package directories,
+      all of these are added recursively.  *basename* is intended for internal
+      use only.  The :meth:`writepy` method makes archives with file names like
+      this::
+
+         string.pyc                   # Top level name
+         test/__init__.pyc            # Package directory
+         test/testall.pyc             # Module test.testall
+         test/bogus/__init__.pyc      # Subpackage directory
+         test/bogus/myfile.pyc        # Submodule test.bogus.myfile
 
 
 .. _zipinfo-objects:

Include/compile.h

@@ -29,8 +29,9 @@ typedef struct {
 #define FUTURE_BARRY_AS_BDFL "barry_as_FLUFL"
 
 struct _mod; /* Declare the existence of this type */
-PyAPI_FUNC(PyCodeObject *) PyAST_Compile(struct _mod *, const char *,
-					PyCompilerFlags *, PyArena *);
+#define PyAST_Compile(mod, s, f, ar) PyAST_CompileEx(mod, s, f, -1, ar)
+PyAPI_FUNC(PyCodeObject *) PyAST_CompileEx(struct _mod *, const char *,
+					   PyCompilerFlags *, int, PyArena *);
 PyAPI_FUNC(PyFutureFeatures *) PyFuture_FromAST(struct _mod *, const char *);
 
 

Include/pythonrun.h

@@ -76,9 +76,10 @@ PyAPI_FUNC(PyObject *) PyRun_FileExFlags(FILE *, const char *, int,
 #ifdef Py_LIMITED_API
 PyAPI_FUNC(PyObject *) Py_CompileStringFlags(const char *, const char *, int);
 #else
-#define Py_CompileString(str, p, s) Py_CompileStringFlags(str, p, s, NULL)
-PyAPI_FUNC(PyObject *) Py_CompileStringFlags(const char *, const char *, int,
-                                             PyCompilerFlags *);
+#define Py_CompileString(str, p, s) Py_CompileStringExFlags(str, p, s, NULL, -1)
+#define Py_CompileStringFlags(str, p, s, f) Py_CompileStringExFlags(str, p, s, f, -1)
+PyAPI_FUNC(PyObject *) Py_CompileStringExFlags(const char *, const char *, int,
+                                               PyCompilerFlags *, int);
 #endif
 PyAPI_FUNC(struct symtable *) Py_SymtableString(const char *, const char *, int);
 

Lib/compileall.py

@@ -19,8 +19,8 @@
 
 __all__ = ["compile_dir","compile_file","compile_path"]
 
-def compile_dir(dir, maxlevels=10, ddir=None,
-                force=False, rx=None, quiet=False, legacy=False):
+def compile_dir(dir, maxlevels=10, ddir=None, force=False, rx=None,
+                quiet=False, legacy=False, optimize=-1):
     """Byte-compile all modules in the given directory tree.
 
     Arguments (only dir is required):
@@ -32,6 +32,7 @@ def compile_dir(dir, maxlevels=10, ddir=None,
     force:     if True, force compilation, even if timestamps are up-to-date
     quiet:     if True, be quiet during compilation
     legacy:    if True, produce legacy pyc paths instead of PEP 3147 paths
+    optimize:  optimization level or -1 for level of the interpreter
     """
     if not quiet:
         print('Listing', dir, '...')
@@ -51,7 +52,8 @@ def compile_dir(dir, maxlevels=10, ddir=None,
         else:
             dfile = None
         if not os.path.isdir(fullname):
-            if not compile_file(fullname, ddir, force, rx, quiet, legacy):
+            if not compile_file(fullname, ddir, force, rx, quiet,
+                                legacy, optimize):
                 success = 0
         elif (maxlevels > 0 and name != os.curdir and name != os.pardir and
               os.path.isdir(fullname) and not os.path.islink(fullname)):
@@ -61,14 +63,15 @@ def compile_dir(dir, maxlevels=10, ddir=None,
     return success
 
 def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
-                 legacy=False):
+                 legacy=False, optimize=-1):
     """Byte-compile file.
     fullname:  the file to byte-compile
     ddir:      if given, purported directory name (this is the
                directory name that will show up in error messages)
     force:     if True, force compilation, even if timestamps are up-to-date
     quiet:     if True, be quiet during compilation
     legacy:    if True, produce legacy pyc paths instead of PEP 3147 paths
+    optimize:  optimization level or -1 for level of the interpreter
     """
     success = 1
     name = os.path.basename(fullname)
@@ -84,7 +87,11 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
         if legacy:
             cfile = fullname + ('c' if __debug__ else 'o')
         else:
-            cfile = imp.cache_from_source(fullname)
+            if optimize >= 0:
+                cfile = imp.cache_from_source(fullname,
+                                              debug_override=not optimize)
+            else:
+                cfile = imp.cache_from_source(fullname)
             cache_dir = os.path.dirname(cfile)
         head, tail = name[:-3], name[-3:]
         if tail == '.py':
@@ -101,7 +108,8 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
             if not quiet:
                 print('Compiling', fullname, '...')
             try:
-                ok = py_compile.compile(fullname, cfile, dfile, True)
+                ok = py_compile.compile(fullname, cfile, dfile, True,
+                                        optimize=optimize)
             except py_compile.PyCompileError as err:
                 if quiet:
                     print('*** Error compiling', fullname, '...')
@@ -126,7 +134,7 @@ def compile_file(fullname, ddir=None, force=0, rx=None, quiet=False,
     return success
 
 def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
-                 legacy=False):
+                 legacy=False, optimize=-1):
     """Byte-compile all module on sys.path.
 
     Arguments (all optional):
@@ -136,6 +144,7 @@ def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
     force: as for compile_dir() (default False)
     quiet: as for compile_dir() (default False)
     legacy: as for compile_dir() (default False)
+    optimize: as for compile_dir() (default -1)
     """
     success = 1
     for dir in sys.path:
@@ -144,7 +153,7 @@ def compile_path(skip_curdir=1, maxlevels=0, force=False, quiet=False,
         else:
             success = success and compile_dir(dir, maxlevels, None,
                                               force, quiet=quiet,
-                                              legacy=legacy)
+                                              legacy=legacy, optimize=optimize)
     return success
 
 

Lib/py_compile.py

@@ -72,7 +72,7 @@ def wr_long(f, x):
                    (x >> 16) & 0xff,
                    (x >> 24) & 0xff]))
 
-def compile(file, cfile=None, dfile=None, doraise=False):
+def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1):
     """Byte-compile one Python source file to Python bytecode.
 
     :param file: The source file name.
@@ -86,6 +86,10 @@ def compile(file, cfile=None, dfile=None, doraise=False):
         will be printed, and the function will return to the caller. If an
         exception occurs and this flag is set to True, a PyCompileError
         exception will be raised.
+    :param optimize: The optimization level for the compiler.  Valid values
+        are -1, 0, 1 and 2.  A value of -1 means to use the optimization
+        level of the current interpreter, as given by -O command line options.
+
     :return: Path to the resulting byte compiled file.
 
     Note that it isn't necessary to byte-compile Python modules for
@@ -111,7 +115,8 @@ def compile(file, cfile=None, dfile=None, doraise=False):
             timestamp = int(os.stat(file).st_mtime)
         codestring = f.read()
     try:
-        codeobject = builtins.compile(codestring, dfile or file,'exec')
+        codeobject = builtins.compile(codestring, dfile or file, 'exec',
+                                      optimize=optimize)
     except Exception as err:
         py_exc = PyCompileError(err.__class__, err, dfile or file)
         if doraise:
@@ -120,7 +125,10 @@ def compile(file, cfile=None, dfile=None, doraise=False):
             sys.stderr.write(py_exc.msg + '\n')
             return
     if cfile is None:
-        cfile = imp.cache_from_source(file)
+        if optimize >= 0:
+            cfile = imp.cache_from_source(file, debug_override=not optimize)
+        else:
+            cfile = imp.cache_from_source(file)
     try:
         os.makedirs(os.path.dirname(cfile))
     except OSError as error: