======================
QA checks and warnings
======================

.. highlight:: text

This section explains Python-related QA checks and the resulting QA
warnings that can be output while running the package manager or related
tooling.


Improved QA warning reporting in Portage
========================================
Normally, Portage outputs QA warnings at specific phases of the build
process.  They are usually interspersed with other verbose output,
and they are easy to miss, especially when building multiple packages
in a single batch.

To make them easier to catch, Portage's elog system can be used
to repeat all the QA warnings once emerge exits.  The required "echo"
module is already enabled by default, however it skips QA warnings
by default.  To change that, set in your ``make.conf``:

.. code-block:: bash

    PORTAGE_ELOG_CLASSES="log warn error qa"

For more information on using Portage's elog system, please refer
to make.conf.example_ included in the Portage distribution.


Compiled bytecode-related warnings
==================================
To improve performance, the Python interpreter compiles Python sources
into bytecode.  CPython and PyPy3 feature three optimization levels
that impact the bytecode size:

1. no optimizations (the default)

2. ``-O`` that removes assert statements from code

3. ``-OO`` that removes assert statements and docstrings

Normally, the compiled bytecode is stored on disk in ``.pyc`` files
(historically, Python 2 has also used ``.pyo`` files).
When these files are present and up-to-date, Python loads
the precompiled bytecode from them rather than creating it from
the source code, improving module loading time.  When they are missing,
Python normally creates them automatically if it has write permissions
to the respective directory.

In Gentoo, we aim for packages to always byte-compile all their Python
files to all supported optimization levels.  Besides improving module
loading performance, it ensures that the Python interpreters will not
attempt to write compiled bytecode themselves, effectively creating
files that are not monitored by the package manager (and therefore e.g.
are not removed when the respective package is uninstalled) or causing
sandbox violations while building other packages.

The Gentoo repository features a QA check to ensure that all installed
Python modules are byte-compiled for all optimization levels supported
by the respective Python interpreter, and that no stray compiled files
exist.  This check is implemented using an auxiliary command found
in ``app-portage/gpep517-7`` and newer.  This check can also be run
manually (with machine-readable output) using e.g.::

    $ python3.10 -m gpep517 verify-pyc --destdir=/tmp/portage/dev-python/trimesh-3.12.9/image
    missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py
    missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py
    missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py


Modules are not byte-compiled
-----------------------------
The most common QA warning that can be noticed while building packages
indicates that at least some of the expected ``.pyc`` files are missing.
For example::

     * QA Notice: This package installs one or more Python modules that are
     * not byte-compiled.
     * The following files are missing:
     *
     *   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc
     *   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc
     *   /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc

     * QA Notice: This package installs one or more Python modules that are
     * not byte-compiled.
     * The following files are missing:
     *
     *   /usr/lib/python3.10/site-packages/blueman/__pycache__/Constants.cpython-310.opt-2.pyc
     *   /usr/lib/python3.10/site-packages/blueman/__pycache__/DeviceClass.cpython-310.opt-2.pyc
     *   /usr/lib/python3.10/site-packages/blueman/__pycache__/Functions.cpython-310.opt-2.pyc
     *   /usr/lib/python3.10/site-packages/blueman/__pycache__/Sdp.cpython-310.opt-2.pyc
     [...]

There are three common causes for these warnings:

1. The package's build system nor the ebuild do not byte-compile
   installed Python modules — the warning lists all optimization levels
   for all installed modules.

2. The package's build system byte-compiles installed modules only for
   a subset of optimization levels — the warning lists all modules
   but only for a subset of levels (the second example in the example
   snippet).

3. The package installs ``.py`` files with incorrect syntax that can not
   be byte-compiled and usually trigger syntax errors during the install
   phase (the first example in the above snippet).

For the first two cases, the better solution is to patch the respective
build system to perform byte-compilation for all optimization levels.
An acceptable workaround is to call ``python_optimize`` from the ebuild
(note that in PEP517 mode, distutils-r1 does that unconditionally).

For the third case, the only real solution is to submit a fix upstream
that renames files that do not contain valid Python modules to use
another suffix.  For example, the template triggering the QA warning
in trimesh package could be renamed from ``.py`` to ``.py.tmpl``.


Stray compiled bytecode
-----------------------
The following QA warning indicates that there are stray ``.pyc`` files
that are not clearly matching any installed Python module-implementation
pair::

     * QA Notice: This package installs one or more compiled Python modules
     * that do not match installed modules (or their implementation).
     * The following files are stray:
     *
     *   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-35.pyc
     *   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-36.pyc
     *   /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-38.pyc

There are two common causes for this:

1. The package is shipping precompiled ``.pyc`` files and installing
   them along with ``.py`` modules.  The ebuild should remove the stray
   files in ``src_prepare`` then.

2. The ebuild is attempting to remove some ``.py`` files after they have
   been byte-compiled.  It needs to be modified to either remove them
   prior to the byte-compilation stage, or to fix the build system
   not to install them in the first place.


Stray top-level files in site-packages
======================================
distutils-r1 checks for the common mistake of installing unexpected
files that are installed top-level into the site-packages directory.
An example error due to that looks like the following::

     * The following unexpected files/directories were found top-level
     * in the site-packages directory:
     *
     *   /usr/lib/python3.10/site-packages/README.md
     *   /usr/lib/python3.10/site-packages/LICENSE
     *   /usr/lib/python3.10/site-packages/CHANGELOG
     *
     * This is most likely a bug in the build system.  More information
     * can be found in the Python Guide:
     * https://projects.gentoo.org/python/guide/qawarn.html#stray-top-level-files-in-site-packages

In general, it is desirable to prepare a fix for the build system
and submit it upstream.  However, it is acceptable to remove the files
locally in the ebuild while waiting for a release with the fix.

The subsequent sections describe the common causes and the suggested
fixes.


Example for test packages installed by setuptools
-------------------------------------------------
Many packages using the setuptools build system utilize the convenient
``find_packages()`` method to locate the Python sources.  In some cases,
this method also wrongly grabs top-level test directories or other files
that were not intended to be installed.

For example, the following invocation will install everything that looks
like a Python package from the source tree:

.. code-block:: python

    setup(
        packages=find_packages())

The correct fix for this problem is to add an ``exclude`` parameter
that restricts the installed package list, for example:

.. code-block:: python

    setup(
        packages=find_packages(exclude=["tests", "tests.*"]))

Note that if the top-level ``tests`` package has any subpackages, both
``tests`` and ``tests.*`` need to be listed.

If ``setup.cfg`` is used instead, the excludes are specified as follows:

.. code-block:: ini

    [options.packages.find]
    exclude =
        tests
        tests.*

If ``pyproject.toml`` is used:

.. code-block:: toml

    [tool.setuptools.packages.find]
    exclude = [
        "tests",
        "tests.*",
    ]

For reference, see `custom discovery in setuptools documentation`_.


Documentation files installed by Poetry
---------------------------------------
It is a relatively common problem that packages using the Poetry build
system are installing documentation files (such as ``README``)
to the site-packages directory.  This is because of incorrect
``include`` use in ``pyproject.toml``.  For example, consider
the following configuration:

.. code-block:: toml

    include = [
        "CHANGELOG",
        "README.md",
        "LICENSE"
    ]

The author meant to include these files in the source distribution
packages.  However, the ``include`` key applies to wheels as well,
effectively including them in files installed into ``site-packages``.

To fix that, you need to specify file formats explicitly, for every
entry:

.. code-block:: toml

    include = [
        { path = "CHANGELOG", format = "sdist" },
        { path = "README.md", format = "sdist" },
        { path = "LICENSE", format = "sdist" },
    ]

For reference, see `include and exclude in Poetry documentation`_.


Deprecated PEP 517 backends
===========================

flit.buildapi
-------------
Some packages are still found using the historical flit build backend.
Their ``pyproject.toml`` files contain a section similar
to the following:

.. code-block:: toml

    [build-system]
    requires = ["flit"]
    build-backend = "flit.buildapi"

This backend requires installing the complete flit package manager.
Instead, the package should be fixed upstream to use flit_core
per `flit build system section documentation`_ instead:

.. code-block:: toml

    [build-system]
    requires = ["flit_core"]
    build-backend = "flit_core.buildapi"

flit_core produces identical artifacts to flit.  At the same time, it
reduces the build-time dependency footprint and therefore makes isolated
PEP 517 builds faster.


poetry.masonry.api
------------------
A similar problem applies to the packages using poetry.  The respective
``pyproject.toml`` files contain:

.. code-block:: toml

    [build-system]
    requires = ["poetry>=0.12"]
    build-backend = "poetry.masonry.api"

Instead, the lightweight poetry-core module should be used per `poetry
PEP-517 documentation`_:

.. code-block:: toml

    [build-system]
    requires = ["poetry_core>=1.0.0"]
    build-backend = "poetry.core.masonry.api"

poetry-core produces identical artifacts to poetry.  It has smaller
dependency footprint and makes isolated builds much faster.


setuptools.build_meta:__legacy__
--------------------------------
Some packages using setuptools specify the following:

.. code-block:: toml

    [build-system]
    requires = ["setuptools>=40.8.0", "wheel"]
    build-backend = "setuptools.build_meta:__legacy__"

This is incorrect, as the legacy backend is intended to be used only
as an implicit fallback.  All packages should be using the regular
backend instead:

.. code-block:: toml

    [build-system]
    requires = ["setuptools>=40.8.0"]
    build-backend = "setuptools.build_meta"

Please also note that the ``wheel`` package should *not* be listed
as a dependency, as it is an implementation detail and it was always
implicitly returned by the backend.  Unfortunately, due to prolonged
documentation error, a very large number of packages still specifies it,
and other packages tend to copy that mistake.


.. _make.conf.example:
   https://gitweb.gentoo.org/proj/portage.git/tree/cnf/make.conf.example#n330
.. _custom discovery in setuptools documentation:
   https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#custom-discovery
.. _include and exclude in Poetry documentation:
   https://python-poetry.org/docs/pyproject/#include-and-exclude
.. _flit build system section documentation:
   https://flit.readthedocs.io/en/latest/pyproject_toml.html#build-system-section
.. _poetry PEP-517 documentation:
   https://python-poetry.org/docs/pyproject/#poetry-and-pep-517