ci: add downstream testing

[henryiii]

This adds a downstream testing job. Three packages so far: packaging_legacy, build, and pyproject-metadata.

[henryiii]

Dropping this here, trying to add hatch:

index 692dfa7..2c2c083 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -81,6 +81,7 @@ PROJECTS = {
     "packaging_legacy": "https://github.com/di/packaging_legacy/archive/refs/tags/23.0.post0.tar.gz",
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
+    "hatch": "https://github.com/pypa/hatch/archive/refs/tags/hatch-v1.16.2.tar.gz",
 }


@@ -108,6 +109,9 @@ def downstream(session: nox.Session, project: str) -> None:
         session.install("-e.")
         session.run(*pip_cmd, "list")
         session.run("pytest", env=env)
+    elif project == "hatch":
+        session.install("-e.", "pytest", "filelock", "flit-core", "trustme", "editables")
+        session.run("pytest", env=env)
     elif project in {"build", "pyproject_metadata"}:
         session.install("-e.", "-

There still is a known breakage (@ spacing changed), and there are some breakages I think because it expects to run from the git root.

I tried pip too:

    elif project == "pip":
        session.install("-e.", "--group=test")
        session.run("pip", "wheel", "-w", "tests/data/common_wheels", "--group", "test-common-wheels")
        session.run(*pip_cmd, "list")
        session.run("pytest")

but the test suite is quite slow. I didn't get to setting up vendoring to replace its copy of packaging. Another candidate for downstream testing, pip-tools, uses pip._vendor.packaging. Poetry-core vendors as well.

[henryiii]

Suggested change

	name: Downstream
	name: Downstream ${{ matrix.project }}

[brettcannon]

trying to add hatch:

Did you want me to point Ofek at this issue?

[henryiii]

You can, there's also pypa/hatch#2159, which is related.

[abravalheri]

(FYI: I am letting the CI run some tests for setuptools: pypa/setuptools#5155, hopefully it will be fine)

[henryiii]

Great, I just tried that here too; the only downside is it's pretty slow (several minutes), but I can make it work. Added a print inside our code to verify setuptools is picking it up when I patch it in, it's working locally.

diff --git a/noxfile.py b/noxfile.py
index 692dfa7..ed19c1d 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -80,6 +80,7 @@ def tests(session: nox.Session) -> None:
 PROJECTS = {
     "packaging_legacy": "https://github.com/di/packaging_legacy/archive/refs/tags/23.0.post0.tar.gz",
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
+    "setuptools": "https://github.com/pypa/setuptools/archive/refs/tags/v80.10.1.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
 }

@@ -87,6 +88,7 @@ PROJECTS = {
 @nox.parametrize("project", list(PROJECTS))
 @nox.session(default=False)
 def downstream(session: nox.Session, project: str) -> None:
+    pkg_dir = Path.cwd() / "src/packaging"
     env = {"FORCE_COLOR": None}
     session.install("-e.")

@@ -107,12 +109,19 @@ def downstream(session: nox.Session, project: str) -> None:
         session.install("-r", "tests/requirements.txt")
         session.install("-e.")
         session.run(*pip_cmd, "list")
-        session.run("pytest", env=env)
+        session.run("pytest", *session.posargs, env=env)
     elif project in {"build", "pyproject_metadata"}:
         session.install("-e.", "--group=test")
         if project != "build":
             session.run(*pip_cmd, "list")
-        session.run("pytest", env=env)
+        session.run("pytest", *session.posargs, env=env)
+    elif project == "setuptools":
+        session.install("-e.[test,cover]")
+        session.run(*pip_cmd, "list")
+        repl_dir = "setuptools/_vendor/packaging"
+        shutil.rmtree(repl_dir)
+        shutil.copytree(pkg_dir, repl_dir)
+        session.run("pytest", *session.posargs, env=env)
     else:
         session.error("Unknown package")

[henryiii]

I can run pip using the same idea too, but the test suite takes 53 minutes on my Intel machine. I see the spacing changes, so it's using the right version of packaging.

diff --git a/noxfile.py b/noxfile.py
index ed19c1d..658b806 100644
--- a/noxfile.py
+++ b/noxfile.py
@@ -82,6 +82,7 @@ PROJECTS = {
     "build": "https://github.com/pypa/build/archive/refs/tags/1.4.0.tar.gz",
     "setuptools": "https://github.com/pypa/setuptools/archive/refs/tags/v80.10.1.tar.gz",
     "pyproject_metadata": "https://github.com/pypa/pyproject-metadata/archive/refs/tags/0.10.0.tar.gz",
+    "pip": "https://github.com/pypa/pip/archive/refs/tags/25.3.tar.gz",
 }


@@ -122,6 +123,14 @@ def downstream(session: nox.Session, project: str) -> None:
         shutil.rmtree(repl_dir)
         shutil.copytree(pkg_dir, repl_dir)
         session.run("pytest", *session.posargs, env=env)
+    elif project == "pip":
+        session.install("-e.", "--group=test")
+        session.run("pip", "wheel", "-w", "tests/data/common_wheels", "--group", "test-common-wheels")
+        session.run(*pip_cmd, "list")
+        repl_dir = "src/pip/_vendor/packaging"
+        shutil.rmtree(repl_dir)
+        shutil.copytree(pkg_dir, repl_dir)
+        session.run("pytest")
     else:
         session.error("Unknown package")

[notatallshaw]

I would be happy to work on some useful subset of pip unit and feature tests most likely impacted by packaging changes, but I'd be surprised if we got it to less than 10% the full runtime.

[abravalheri]

the only downside is it's pretty slow (several minutes)

Yeah, our tests are pretty slow 😅.
If you prefer to not have it coded in the packaging test suite, just give me a heads up and we can always run the setuptools CI with the beta release of packaging.

I tested simply creating a PR on setuptools with the newer dependency listed in pyproject.toml and it seems to suffice... (unless I am missing something)