Skip to content

PEP 770: Add the dist-info.files table and reserve dist-info dirs #4283

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brettcannon merged 6 commits into from
Mar 4, 2025
Merged

PEP 770: Add the dist-info.files table and reserve dist-info dirs #4283

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
5946028
PEP 770: Add the additional-files table and reserve dist-info dirs
sethmlarson Feb 25, 2025
a307eb4
Address review comments
sethmlarson Feb 26, 2025
037d2fb
Finish acknowledgement
sethmlarson Feb 26, 2025
f0258e8
Address review comments
sethmlarson Feb 26, 2025
502496b
Address more review comments
sethmlarson Feb 27, 2025
2afcfa0
Rename new table to 'dist-info.files'
sethmlarson Feb 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Next Next commit
PEP 770: Add the additional-files table and reserve dist-info dirs
  • Loading branch information
@sethmlarson
sethmlarson committed Feb 25, 2025
commit 59460282922f88769d35ecdba67c7645a5bb3210
293 changes: 158 additions & 135 deletions peps/pep-0770.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,98 +179,58 @@ Specification

The changes necessary to implement this PEP include:

* Additions to `Core Metadata <770-spec-core-metadata_>`_, as defined in the
`Core Metadata specification <coremetadataspec_>`__.
* Additions to the author-provided
`project source metadata <770-spec-project-source-metadata_>`_, as defined in the
`pyproject.toml specification <pyprojecttoml_>`__.
* `Additions <770-spec-project-formats_>`_ to the source distribution (sdist),
built distribution (wheel), and installed project specifications.
* A new reserved registry of subdirectory names in the ``.dist-info`` directory.
Comment thread
sethmlarson marked this conversation as resolved.
Outdated
* A new reserved optional ``[additional-files]`` table with an optional
``sboms`` key added to
`project source metadata <770-spec-project-source-metadata_>`_,
Comment thread
sethmlarson marked this conversation as resolved.
Outdated
as defined in the `pyproject.toml specification <pyprojecttoml_>`__.
* `Additions <770-spec-project-formats_>`_ to the built distribution (wheel),
and installed project specifications

In addition to the above, an informational PEP will be created for tools
consuming included SBOM documents and other Python package metadata to
generate complete SBOM documents for Python packages.

Terminology
-----------
.. _770-spec-dist-info-subdirs:

Reserved ``.dist-info`` subdirectories registry
-----------------------------------------------

This PEP introduces a new registry of reserved subdirectory names allowed in
the ``.dist-info`` directory for the :term:`distribution archive`
and :term:`installed project` s project types. Future additions to this registry
will be made through the PEP process. The initial values in this registry are:

.. table::
Comment thread
sethmlarson marked this conversation as resolved.
Outdated

This section describes terminology used later in the document:

.. glossary::

root SBOM directory
The directory under which SBOM files are stored in a
:term:`project source tree`, :term:`distribution archive`
or :term:`installed project`.
Also, the root directory that their paths
recorded in the :ref:`Sbom-File <770-spec-sbom-file-field>`
:term:`Core Metadata field` are relative to.
Defined to be the :term:`project root directory`
for a :term:`project source tree` or
:term:`source distribution <Source Distribution (or "sdist")>`;
and a subdirectory named ``sboms`` of
the directory containing the :term:`built metadata`—
i.e., the ``.dist-info/sboms`` directory—
for a :term:`Built Distribution` or :term:`installed project`.

.. _770-spec-core-metadata:

Core Metadata
-------------

.. _770-spec-sbom-file-field:

Add ``Sbom-File`` field
~~~~~~~~~~~~~~~~~~~~~~~

The ``Sbom-File`` is a new optional Core Metadata field. Each instance contains a
string representation of the path to an SBOM document. The path is specified
relative to the :term:`root SBOM directory` for all project types. It is a
multi-use field that MAY appear zero or more times and each instance lists the
path to one such file. Files specified under this field are SBOM documents
that are distributed with the package.

As `specified by this PEP <#770-spec-project-formats>`__, its value is also
that file's path relative to the :term:`root SBOM directory` in both installed
projects and the standardized Distribution Package types.

If an ``Sbom-File`` is listed in a
:term:`Source Distribution <Source Distribution (or "sdist")>` or
:term:`Built Distribution`'s Core Metadata:

* That file MUST be included in the :term:`distribution archive` at the
specified path relative to the :term:`root SBOM directory`.
* Installers MUST install the file with the :term:`project` at that same
relative path.
* Inside the :term:`root SBOM directory`, packaging tools MUST reproduce the
directory structure under which the source files are located relative to the
project root.
* Path delimiters MUST be the forward slash character (``/``), and parent
directory indicators (``..``) MUST NOT be used.

For all newly-uploaded distribution archives that include one or more
``Sbom-File`` fields in their Core Metadata and declare a ``Metadata-Version``
of ``2.5`` or higher, PyPI and other indices SHOULD validate that all files
specified with ``Sbom-File`` are present in the distribution archives.
================= ==========
Directory name PEP
================= ==========
``licenses`` :pep:`639`
Comment thread
sethmlarson marked this conversation as resolved.
Outdated
``license_files`` N/A (See :ref:`770-backwards-compat`)
``sboms`` :pep:`770`
================= ==========

Build backends MUST NOT create subdirectories in the ``.dist-info`` directory
beyond the names in the registry to avoid collisions with future reserved names.

Comment thread
sethmlarson marked this conversation as resolved.
.. _770-spec-project-source-metadata:

Project source metadata
-----------------------

This PEP specifies changes to the project's source metadata under a
``[project]`` table in the ``pyproject.toml`` file.
This PEP specifies changes to the project's source metadata
in the ``pyproject.toml`` file:

Add ``sbom-files`` key
~~~~~~~~~~~~~~~~~~~~~~
Add new ``[additional-files]`` table
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A new optional ``sbom-files`` key is added to the ``[project]`` table for
specifying paths in the project source tree relative to ``pyproject.toml`` to
file(s) containing SBOMs to be distributed with the package. This key
corresponds to the ``Sbom-File`` fields in the Core Metadata.
A new optional ``[additional-files]`` table is added for specifying paths
in the project source tree relative to ``pyproject.toml`` to file(s) which
should be included in the built project to a defined directory.

Its value is an array of strings which MUST contain valid glob patterns, as
specified below:
This new table has only one defined optional key: ``sboms``. The value of the
``sboms`` key MUST be an array of valid glob patterns, as specified below:

* Alphanumeric characters, underscores (``_``), hyphens (``-``) and dots (``.``)
MUST be matched verbatim.
Expand All @@ -292,50 +252,48 @@ they can also be defined.

Build tools:

* MUST treat each value as a glob pattern, and MUST raise an error if the
pattern contains invalid glob syntax.
* MUST treat each value in the array as a glob pattern, and MUST raise an error
if the pattern contains invalid glob syntax.
* MUST include all files matched by a listed pattern in all distribution
archives.
* MUST list each matched file path under an ``Sbom-File`` field in the
Core Metadata.
archives under the ``.dist-info/sboms`` directory.
* MUST raise an error if any individual user-specified pattern does not match
at least one file.

If the ``sbom-files`` key is present and is set to a value of an empty array,
If the ``sboms`` key is present and is set to a value of an empty array,
then tools MUST NOT include any SBOM files and MUST NOT raise an error.

Examples of valid SBOM files declarations:

.. code-block:: toml

[project]
sbom-files = ["bom.json"]
[additional-files]
sboms = ["bom.json"]

[project]
sbom-files = ["sboms/openssl.cdx.json", "sboms/openssl.spdx.json"]
[additional-files]
sboms = ["sboms/openssl.cdx.json", "sboms/openssl.spdx.json"]

[project]
sbom-files = ["sboms/*"]
[additional-files]
sboms = ["sboms/*"]

[project]
sbom-files = []
[additional-files]
sboms = []

Examples of invalid SBOM files declarations:

.. code-block:: toml

[project]
sbom-files = ["..\bom.json"]
[additional-files]
sboms = ["..\bom.json"]

Reason: ``..`` must not be used. ``\\`` is an invalid path delimiter, ``/``
must be used.

.. code-block:: toml

[project]
sbom-files = ["bom{.json*"]
[additional-files]
sboms = ["bom{.json*"]

Reason: ``bom{.json`` is not a valid glob.
Reason: ``bom{.json*`` is not a valid glob.

.. _770-spec-project-formats:

Expand All @@ -345,36 +303,22 @@ SBOM files in project formats
A few additions will be made to the existing specifications.

:term:`Project source trees <Project source tree>`
Per :ref:`639-spec-source-metadata` section, the
Per :ref:`770-spec-project-source-metadata` section, the
`Declaring Project Metadata specification <pyprojecttoml_>`__
will be updated to reflect that SBOM file paths MUST be relative to the
project root directory; i.e. the directory containing the ``pyproject.toml``
(or equivalently, other legacy project configuration,
e.g. ``setup.py``, ``setup.cfg``, etc).

:term:`Source distributions (sdists) <Source Distribution (or "sdist")>`
The sdist specification will be updated to reflect that if the
``Metadata-Version`` is ``2.5`` or greater, the sdist MUST contain any SBOM
files specified by the ``Sbom-File`` field in the ``PKG-INFO`` at their
respective paths relative to the sdist (containing the ``pyproject.toml`` and
the ``PKG-INFO`` Core Metadata).
will be updated to add the ``[additional-files]`` table
and optional ``sboms`` key.

:term:`Built distributions <Built distribution>` (:term:`wheels <wheel>`)
The wheel specification will be updated to reflect that if the
``Metadata-Version`` is ``2.5`` or greater and one or more ``Sbom-File``
fields are specified, the ``.dist-info`` directory MUST contain an ``sboms``
subdirectory, which MUST contain the files listed in the ``Sbom-File`` fields
in the ``METADATA`` file at their respective paths relative to the ``sboms``
directory.

The wheel specification will be updated to add the new registry of reserved
directory names and to reflect that if the ``.dist-info/sboms`` subdirectory
is specified that the directory contains SBOM files.

:term:`Installed projects <Installed project>`
The Recording Installed Projects specification will be updated to reflect that
if the ``Metadata-Version`` is ``2.5`` or greater and one or more
``Sbom-File`` fields is specified, the ``.dist-info`` directory MUST contain
an ``sboms`` subdirectory which MUST contain the files listed in the
``Sbom-File`` fields in the ``METADATA`` file at their respective paths
relative to the ``sboms`` directory, and that any files in this directory MUST
be copied from wheels by install tools.
The Recording Installed Projects specification will be updated to reflect
that if the ``.dist-info/sboms`` subdirectory is specified that the directory
contains SBOM files and that any files in this directory MUST be copied from
wheels by install tools.

SBOM data interoperability
--------------------------
Expand Down Expand Up @@ -406,18 +350,69 @@ PyPI and other indices MAY validate the contents of SBOM documents specified by
this PEP, but MUST NOT validate or reject data for unknown
SBOM standards, versions, or fields.

.. _770-backwards-compat:

Backwards Compatibility
=======================

There are no backwards compatibility concerns for this PEP.

The changes to Python package Core Metadata and ``pyproject.toml`` are
only additive, this PEP doesn't change the behavior of any existing fields.

Tools which are processing Python packages can use the ``Sbom-File`` core
metadata field to clearly delineate between packages which include SBOM
documents that implement this PEP (and thus have more requirements) and
packages which include SBOM documents before this PEP was authored.
Reserved ``.dist-info`` subdirectories registry
-----------------------------------------------

The new registry of reserved ``.dist-info`` subdirectories represents
a new reservation that wasn't previously documented, thus has the potential to
break assumptions being made by already existing tools.

To check what ``.dist-info`` subdirectory names are in use today
a query across
`all files in package archives on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`__
was executed:

.. code-block:: sql

SELECT (
regexp_extract(archive_path, '.*\.dist-info/([^/]+)/', 1) AS dirname,
COUNT(DISTINCT project_name) AS projects
)
FROM '*.parquet'
WHERE archive_path LIKE '%.dist-info/%/%'
GROUP BY dirname ORDER BY projects DESC;

Note that this only includes records for
*files* and thus won't return results for empty directories. Empty directories
being pervasively used and somehow load-bearing is unlikely, so is an accepted
risk of using this method. This query yielded the following results:

.. table::
Comment thread
sethmlarson marked this conversation as resolved.
Outdated

====================== ===============
Subdirectory Unique Projects
====================== ===============
``licenses`` 22,026
``license_files`` 1,828
``LICENSES`` 170
``.ipynb_checkpoints`` 85
``license`` 18
``.wex`` 9
``dist`` 8
``include`` 6
``build`` 5
``tmp`` 4
``src`` 3
``calmjs_artifacts`` 3
``.idea`` 2
====================== ===============
Comment thread
sethmlarson marked this conversation as resolved.
Outdated

Not shown above are around ~50 other subdirectory names that are used in a
single project. From these results we can see:

* Most subdirectories under ``.dist-info`` are to do with licensing,
one of which (``licenses``) is specified by :pep:`639` and other
(``license_files``) which is being used by the Hatch build backend.
Comment thread
sethmlarson marked this conversation as resolved.
Outdated
* The ``sboms`` subdirectory doesn't collide with existing use.
* Other subdirectory names under ``.dist-info`` appear to be either not
widespread or accidental.

As a result of this query
Comment thread
sethmlarson marked this conversation as resolved.
Outdated

Security Implications
=====================
Expand Down Expand Up @@ -463,12 +458,11 @@ For packages which cannot be automatically annotated and if the package author
wishes to provide an SBOM the approach will be to generate or author SBOM files
and then include those files using ``pyproject.toml``:

.. code-block:: toml
.. code-block:: toml

[project]
...
sbom-files = [
"sboms/bom.cdx.json"
[additional-files]
sboms = [
"sboms/bom.cdx.json"
]

For projects manually specifying an SBOM document the challenge will be
Expand Down Expand Up @@ -558,6 +552,31 @@ a single SBOM standard. Tools that use SBOM data today already need to support
multiple formats to handle this situation, so a future standard that updates to
require only one standard would have no effect on downstream SBOM tools.

Using metadata fields to specify SBOM files in archives
-------------------------------------------------------

A previous iteration of this specification used an ``Sbom-File`` metadata
field to specify an SBOM file within a source or binary distribution archive.
This would make the implementation similar to :pep:`639` which uses the
``License-File`` field to enumerate license files in archives.

The primary issue with this approach is that SBOM files can originate from both
static and dynamic sources: like versioned source code, the build backend,
or from tools adding SBOM files after the build has completed (like auditwheel).

Metadata fields must either be static or dynamic, not both. This is
in direct conflict with the best-case scenario for SBOM data: that SBOM files
are added automatically by tools during the build of a Python package without
user-involvement or knowledge. Compare this situation to license files which
are almost always static.

The 639-style approach was ultimately dropped in favor of defining SBOMs simply
by their presence in the ``.dist-info/sboms`` directory and using a new table in
``pyproject.toml`` called ``[additional-files]`` to define SBOMs in source
distributions. This approach allows users to specify static SBOM files while
still empowering build backends and tools to add their own SBOM data without the
static/dynamic conflict.

Open Issues
===========

Expand All @@ -579,6 +598,10 @@ References
wheel and then use an SBOM generation tool Syft to detect the SBOM in the
installed package.

* `Querying every file in every release on PyPI <https://sethmlarson.dev/security-developer-in-residence-weekly-report-18>`_.
The dataset available on `py-code.org <py-code.org>`__ from Tom Forbes was
used to check subdirectory usage in ``.dist-info`` files.

.. _phantom dependency: https://www.endorlabs.com/learn/dependency-resolution-in-python-beware-the-phantom-dependency
.. _coremetadataspec: https://packaging.python.org/specifications/core-metadata
.. _pyprojecttoml: https://packaging.python.org/en/latest/specifications/pyproject-toml/
Expand Down