Add a new packaging.dependency_groups module, based on the existing dependency-groups package

[sirosen]

This was initially raised in the PyPA discord.

The primary components of the dependency-groups package, which implements PEP 735, are added in a new module, packaging.dependency_groups.

The CLI components of dependency-groups are ignored here.

Some changes are made, both to adapt the implementation to fit into packaging and to cleanup issues spotted during this pass:

• dependency-groups raises ValueError directly for several forms of
  bad data. packaging modules prefer their own subclasses of
  ValueError, and the error behaviors are therefore adjusted to match.

• dependency-groups does runtime type checks on several function args
  with isinstance, raising TypeError if the wrong type is provided.
  As packaging modules never seem to do this otherwise, these checks
  are removed, in favor of relying on type annotations to document proper
  usage.

• Some of the annotations in dependency-groups are subtly wrong, using
  types like dict where Mapping is more appropriate. Fixing dict
  and list to Mapping and Sequence generally covers this. Because
  str satisfies Sequence[str], one instance of an explicit check for
  str data is added.

• DependencyGroupInclude is converted from a dataclass to a plain class
  with slots. This seems to match other models in packaging (there are
  no other uses of dataclasses). It is also given a repr, to match the
  new doc page's doctest content.

• The functional interface for dependency-groups, resolve(), is
  renamed to the more verbose resolve_dependency_groups(). This makes
  it less ambiguous when used in a from-import, which is common style for
  packaging usage.

• dependency-groups documents the errors which may be raised by various
  methods, but this has been removed for two reasons. First, this was
  less laborious when plain ValueErrors were raised, and it's now more
  to maintain. Second, it's a maintenance burden to ensure these are
  documented properly and there's no known error path which is missing.

The testsuite from dependency-groups is ported over nearly verbatim.
Some tests need updates to match the above changes, and all need minor updates to match style rules in packaging.

[notatallshaw]

• DependencyGroupInclude is converted from a dataclass to a plain class
  with slots. This seems to match other models in packaging (there are
  no other uses of dataclasses). It is also given a repr, to match the
  new doc page's doctest content.

I think it might make sense to start using dataclasses where it simplifies the code, but dataclasses don't play well with slots until Python 3.10, so at least for performance sensitive code it's best to hold off until Python 3.10 can be required.

[henryiii]

You can add slots to dataclasses before 3.10, you just can't auto-generate them. And when you auto-generate them, it actually makes a new class rather than adding them to the current one (which isn't allowed), so technically adding them manually might be a hair better.

[henryiii]

Is it possible to do exceptiongroups? We have exception groups already I believe¹, and I'll be using them for #847. (Haven't checked the code yet, but just a quick thought).

Footnotes

1. Maybe not, was pretty sure we did, but not seeing it quickly. I have an errors.py I could go ahead and put in, though, if that makes sense here. ↩

[sirosen]

Is it possible to do exceptiongroups?

I take this to her a reference to the error collection pattern I used for duplicate names after normalization.
I wouldn't mind changing it, but I'm not sure if it's an improvement vs one exception which presents whatever data we want in attributes. Should the error have useful attributes instead?

My understanding of exception groups is that the target use cases are capturing and propagating multiple unrelated errors -- that's what the original PEP proposed it for. But maybe the practice has evolved beyond those use cases? I haven't used the feature much but am very willing to learn.

Anyway, I'm happy to make any changes requested, but since I'm a little unclear on this one I'm leaving it as-is at the moment.

[henryiii]

In pyproject-metadata, it's used to collect and present multiple unrelated errors in the pyproject.toml. So if a user makes two mistakes, both can be shown, rather than having to fix one at a time then see what else breaks. They do need to be unrelated, otherwise you can't produce both errors.

For example:

[dependency-groups]
test = ["one"]
Test = ["two"]
dev = ["one"]
Dev = ["two"]
start = { include-group = "end" }
end = { include-group = "start" }
something = {}

I could imaging that throwing a group with exceptions:

ExceptionGroup([
    DuplicateGroupNames("test"),
    DuplicateGroupNames("dev"),
    CyclicDependencyGroup("start -> end -> start"),
    InvalidDependencyGroupObject("something"),
])

If it's something you are interested in, I can separate out the errors.py, which has the backported group and a tool that makes it easier to collect them.

Maybe errors in dependency-groups are too rare for this to matter? But it's good to get the API in if you do want it, as users have to be aware of the exception types. In pyproject-metadata, you have to opt into groups because of the API change, it can't easily be added after an API is out.

[sirosen]

Oh, I see. If we allow all of the categories of errors which may be raised to be grouped, we will want an exception group, yes.

I don't know if it's practical to do that for these errors? For example, consider this data:

[dependency-groups]
# a cycle
start = [{ include-group = "end" }]
end = [{ include-group = "start" }]
# "fix" the cycle with a duplicate name
START = []

The current implementation emits DuplicateGroupNames("Duplicate dependency group names: start (start, START)"), and aborts without trying to do any further work.

If we wanted to proceed and find the cycle as well, we'd have to do branched evaluation both against start and START, wouldn't we? I don't like the idea of making the implementation slower and more complex only for that purpose.

But in general, I like collecting up multiple errors and reporting them back when doing so doesn't impose significant extra burdens. I could easily see raising, for your original example,

ExceptionGroup([
    DuplicateGroupNames("test (test, Test)"),
    DuplicateGroupNames("dev (dev, Dev)"),
])

Is that worth doing on its own? We could also maybe group InvalidDependencyGroupObject errors, for data like

[dependency-groups]
# neither element is valid
foo = [{}, {include-groups = ["bar", "baz"]}]

---

I'm generally inclined towards the simpler fail-fast behavior, but don't want to tie our hands WRT future enhancements.

[henryiii]

Reporting as many errors as possible saves human debugging time, so it's worth a little processing time; I'm not sure I know of a situation where parsing these would really need abort-early performance over fully reporting all the errors.

I can try this to see how complex it looks, it's generally manageable with some structure.

[sirosen]

That's fair; I can't think of a performance-critical use case for dependency groups, especially on malformed data.

I think the thing that troubles me is handling duplicate denormed names. Do we really want to expand the full tree of possible combinations?

Here's the sort of data I have in mind:

[dependency-groups]
start = [{include-group = "end"}]
end = [{include-group = "start"}]
START = []
END = []
contains = [{include-group = "start"}]

There are 4 combinations, but the cycle only happens with (start, end).

The only thing I can think to do is to branch, and check for any errors from [(start, end), (start, END), (START, end), (START, END)]. Is that really worthwhile?

If it's acceptable to, in this case, raise an error that start, START and end, END are duplicates, but not detect the cycle, I think the code would be much simpler.

Cases with no duplicate names, like this example, are much easier to handle by "collecting all errors":

[dependency-groups]
start1 = [{include-group = "end1"}]
end1 = [{include-group = "start1"}]

start2 = [{include-group = "end2"}]
end2 = [{include-group = "start2"}]

contains = [{include-group = "start1"}, {include-group = "start2"}]

[henryiii]

I'm not that worried about related errors; you could simply decide an order, and remove them. So if A and a are both groups, you could add the duplicate error, then remove them from the graph and continue. If we then decided later we want to produce more errors (like merging them and then running the graph), that's not an API change like adding the groups in the first place.

[sirosen]

Ah, okay! That makes a lot of sense to me. Expanding the matrix of possibilities seems like more complexity than it's worth, but pruning out duplicates (or just picking one, by some deterministic rule) seems easy.
Let me take a first crack at it.

[henryiii]

Feel free to use errors.py from #847, so we have a shared implementation of ExceptionGroup. :) ErrorCollector is really handy for producing multiple errors, too.

[henryiii]

Ahh, ExceptionGroup is from metadata.py, I knew it was in packaging somewhere already.

I've opened #1071 with errors.py.