Examples

What would this look like in practice?

Let’s say we have a Python package named alpha which contains a sub-package name beta. The source directory layout before byte compilation might look like this:

alpha/
    __init__.py
    one.py
    two.py
    beta/
        __init__.py
        three.py
        four.py

After byte compiling this package with Python 3.2, you would see the following layout:

alpha/
    __pycache__/
        __init__.cpython-32.pyc
        one.cpython-32.pyc
        two.cpython-32.pyc
    __init__.py
    one.py
    two.py
    beta/
        __pycache__/
            __init__.cpython-32.pyc
            three.cpython-32.pyc
            four.cpython-32.pyc
        __init__.py
        three.py
        four.py

Note: listing order may differ depending on the platform.

Let’s say that two new versions of Python are installed, one is Python 3.3 and another is Unladen Swallow. After byte compilation, the file system would look like this:

alpha/
    __pycache__/
        __init__.cpython-32.pyc
        __init__.cpython-33.pyc
        __init__.unladen-10.pyc
        one.cpython-32.pyc
        one.cpython-33.pyc
        one.unladen-10.pyc
        two.cpython-32.pyc
        two.cpython-33.pyc
        two.unladen-10.pyc
    __init__.py
    one.py
    two.py
    beta/
        __pycache__/
            __init__.cpython-32.pyc
            __init__.cpython-33.pyc
            __init__.unladen-10.pyc
            three.cpython-32.pyc
            three.cpython-33.pyc
            three.unladen-10.pyc
            four.cpython-32.pyc
            four.cpython-33.pyc
            four.unladen-10.pyc
        __init__.py
        three.py
        four.py

As you can see, as long as the Python version identifier string is unique, any number of pyc files can co-exist. These identifier strings are described in more detail below.

A nice property of this layout is that the __pycache__ directories can generally be ignored, such that a normal directory listing would show something like this:

alpha/
    __pycache__/
    __init__.py
    one.py
    two.py
    beta/
        __pycache__/
        __init__.py
        three.py
        four.py

This is much less cluttered than even today’s Python.

Case 0: The steady state

When Python is asked to import module foo, it searches for a foo.py file (or foo package, but that’s not important for this discussion) along its sys.path. If found, Python looks to see if there is a matching __pycache__/foo.<magic>.pyc file, and if so, that pyc file is loaded.

Case 1: The first import

When Python locates the foo.py, if the __pycache__/foo.<magic>.pyc file is missing, Python will create it, also creating the __pycache__ directory if necessary. Python will parse and byte compile the foo.py file and save the byte code in __pycache__/foo.<magic>.pyc.

Case 2: The second import

When Python is asked to import module foo a second time (in a different process of course), it will again search for the foo.py file along its sys.path. When Python locates the foo.py file, it looks for a matching __pycache__/foo.<magic>.pyc and finding this, it reads the byte code and continues as usual.

Case 3: __pycache__/foo.<magic>.pyc with no source

It’s possible that the foo.py file somehow got removed, while leaving the cached pyc file still on the file system. If the __pycache__/foo.<magic>.pyc file exists, but the foo.py file used to create it does not, Python will raise an ImportError when asked to import foo. In other words, Python will not import a pyc file from the cache directory unless the source file exists.

Case 4: legacy pyc files and source-less imports

Python will ignore all legacy pyc files when a source file exists next to it. In other words, if a foo.pyc file exists next to the foo.py file, the pyc file will be ignored in all cases

In order to continue to support source-less distributions though, if the source file is missing, Python will import a lone pyc file if it lives where the source file would have been.

Case 5: read-only file systems

When the source lives on a read-only file system, or the __pycache__ directory or pyc file cannot otherwise be written, all the same rules apply. This is also the case when __pycache__ happens to be written with permissions which do not allow for writing containing pyc files.

Detecting PEP 3147 availability

The easiest way to detect whether your version of Python provides PEP 3147 functionality is to do the following check:

>>> import imp
>>> has3147 = hasattr(imp, 'get_tag')

__file__

In Python 3, when you import a module, its __file__ attribute points to its source py file (in Python 2, it points to the pyc file). A package’s __file__ points to the py file for its __init__.py. E.g.:

>>> import foo
>>> foo.__file__
'foo.py'
# baz is a package
>>> import baz
>>> baz.__file__
'baz/__init__.py'

Nothing in this PEP would change the semantics of __file__.

This PEP proposes the addition of an __cached__ attribute to modules, which will always point to the actual pyc file that was read or written. When the environment variable $PYTHONDONTWRITEBYTECODE is set, or the -B option is given, or if the source lives on a read-only filesystem, then the __cached__ attribute will point to the location that the pyc file would have been written to if it didn’t exist. This location of course includes the __pycache__ subdirectory in its path.

For alternative Python implementations which do not support pyc files, the __cached__ attribute may point to whatever information makes sense. E.g. on Jython, this might be the .class file for the module: __pycache__/foo.jython-32.class. Some implementations may use multiple compiled files to create the module, in which case __cached__ may be a tuple. The exact contents of __cached__ are Python implementation specific.

It is recommended that when nothing sensible can be calculated, implementations should set the __cached__ attribute to None.

py_compile and compileall

Python comes with two modules, py_compile 15 and compileall 16 which support compiling Python modules external to the built-in import machinery. py_compile in particular has intimate knowledge of byte compilation, so these will be updated to understand the new layout. The -b flag is added to compileall for writing legacy .pyc byte-compiled file path names.

bdist_wininst and the Windows installer

These tools also compile modules explicitly on installation. If they do not use py_compile and compileall, then they would also have to be modified to understand the new layout.

File extension checks

There exists some code which checks for files ending in .pyc and simply chops off the last character to find the matching .py file. This code will obviously fail once this PEP is implemented.

To support this use case, we’ll add two new methods to the imp package 17:

• imp.cache_from_source(py_path) -> pyc_path
• imp.source_from_cache(pyc_path) -> py_path

Alternative implementations are free to override these functions to return reasonable values based on their own support for this PEP. These methods are allowed to return None when the implementation (or PEP 302 loader 18 in effect) for whatever reason cannot calculate the appropriate file name. They should not raise exceptions.

Backports

For versions of Python earlier than 3.2 (and possibly 2.7), it is possible to backport this PEP. However, in Python 3.2 (and possibly 2.7), this behavior will be turned on by default, and in fact, it will replace the old behavior. Backports will need to support the old layout by default. We suggest supporting PEP 3147 through the use of an environment variable called $PYTHONENABLECACHEDIR or the command line switch -Xenablecachedir to enable the feature.

Makefiles and other dependency tools

Makefiles and other tools which calculate dependencies on .pyc files (e.g. to byte-compile the source if the .pyc is missing) will have to be updated to check the new paths.

Hexadecimal magic tags

pyc files inside of the __pycache__ directories contain a magic tag in their file names. These are mnemonic tags for the actual magic numbers used by the importer. We could have used the hexadecimal representation 10 of the binary magic number as a unique identifier. For example, in Python 3.2:

>>> from binascii import hexlify
>>> from imp import get_magic
>>> 'foo.{}.pyc'.format(hexlify(get_magic()).decode('ascii'))
'foo.580c0d0a.pyc'

This isn’t particularly human friendly though, thus the magic tag proposed in this PEP.

PEP 304

There is some overlap between the goals of this PEP and PEP 304 19, which has been withdrawn. However PEP 304 would allow a user to create a shadow file system hierarchy in which to store pyc files. This concept of a shadow hierarchy for pyc files could be used to satisfy the aims of this PEP. Although the PEP 304 does not indicate why it was withdrawn, shadow directories have a number of problems. The location of the shadow pyc files would not be easily discovered and would depend on the proper and consistent use of the $PYTHONBYTECODE environment variable both by the system and by end users. There are also global implications, meaning that while the system might want to shadow pyc files, users might not want to, but the PEP defines only an all-or-nothing approach.

As an example of the problem, a common (though fragile) Python idiom for locating data files is to do something like this:

from os import dirname, join
import foo.bar
data_file = join(dirname(foo.bar.__file__), 'my.dat')

This would be problematic since foo.bar.__file__ will give the location of the pyc file in the shadow directory, and it may not be possible to find the my.dat file relative to the source directory from there.

Fat byte compilation files

An earlier version of this PEP described “fat” Python byte code files. These files would contain the equivalent of multiple pyc files in a single pyf file, with a lookup table keyed off the appropriate magic number. This was an extensible file format so that the first 5 parallel Python implementations could be supported fairly efficiently, but with extension lookup tables available to scale pyf byte code objects as large as necessary.

The fat byte compilation files were fairly complex, and inherently introduced difficult race conditions, so the current simplification of using directories was suggested. The same problem applies to using zip files as the fat pyc file format.

Multiple file extensions

The PEP author also considered an approach where multiple thin byte compiled files lived in the same place, but used different file extensions to designate the Python version. E.g. foo.pyc25, foo.pyc26, foo.pyc31 etc. This was rejected because of the clutter involved in writing so many different files. The multiple extension approach makes it more difficult (and an ongoing task) to update any tools that are dependent on the file extension.

.pyc

A proposal was floated to call the __pycache__ directory .pyc or some other dot-file name. This would have the effect on *nix systems of hiding the directory. There are many reasons why this was rejected by the BDFL 20 including the fact that dot-files are only special on some platforms, and we actually do not want to hide these completely from users.