RFC: TOML API

[nrdxp]

The API for an atom should be relatively simple, and solve everything we need for it to solve, primarily dependency handling and high-level configuration, e.g. build variants, platforms, etc

Here is a draft

[atom]
id = "mine"
version = "0.1.0"
type = "nix:package" # or config, or chart, or ...

## DEPENENCIES ##

### BUILD TIME ###

#### srcs required for the build
#
[srcs.src] # available at `get.src`
url = "https://example.com/v${major}/${version}/pkg.src.tar.xz"
# capture the version here, for use in the url string
version = "${atom.version}"

#### git sources
#
[srcs.repo]
url = "https://github.com/owner/repo"
ref = "v1"

#### paths
#
[srcs.cargo]
path = "../Cargo.lock" # for use by builder libs

### EVAL TIME ###

#### atoms: the name of the key is the atom id to select from the url
#
[deps.atom] # available at `from.atom`
url = "https://github.com/ekala-project/atom"
version = "^1"

#### > eka add work:mono@^2
#
[deps.my-lib]
url = "https://github.com/org/mono"
version = "^2"

#### local atoms
#
[deps.local]
path = "../../path/to/other/atom"
version = "^3" # version used when resolved remotely

#### allow optional groups?
#
# [deps.lib.a-lib]
# ...
# a list of all the `deps.lib` could then be available at `from.libs`

#### pins have almost the same syntax as `srcs`
#
[pins.pkgs] # available at `from.pkgs`
url = "https://github.com/nixos/nixpkgs"
ref = "nixos-25.05"
# the expression to call `import` on
entry = "pkgs/top-level/impure.nix" 

#### reference values from the toml as arguments
#
[pins.pkgs.args]
localSystem = {system = "${platform.build}"}
crossSystem = {system = "${platform.host}"}

## BUILD CONFIG ##
# unique to the `package` type atom

#### basic platform
#
# platform = "x86_64-linux"

#### supported/tested/cached cross-comp matrix
#
[platform]
build = "x86_64-linux"
host = ["x86_64-linux", "aarch64-linux"]
# target = ...

#### alternate, condensed syntax? [ "builds,", "hosts,", "targets," ]
#
# platforms = [ "x86_64-linux", "{x86_,aarch}64-linux" ] # shell like expansion

#### abstract packages to support variants naturally
# > eka do --cc=clang --host=aarch64-linux <uri>
#
[provide]
ld = ["binutils", "mold"] # from `deps` (priority) or `pins`
cc = ["gcc", "clang"]
libc = ["glibc", "musl"]

# additional distrubition formats to output
# eka get --oci
dist = ["deb", "oci"]

[niklaswimmer]

A few thoughts:

• where is the Nix entrypoint specified? Even if there is a default location, it should be possible to override it
  • does it make sense to have a "type"-specific configuration section?
• how do you want to determine that https://example.com/v${major}/${version}/pkg.src.tar.xz is an http endpoint and https://github.com/org/mono a Git endpoint? I guess heuristics are possible, as well as checking for the presence of ref, but I think the more robust approach would be to use git instead of url as key for Git repositories
• are "builder libs" something built-in, which automatically handle for example Cargo.lock files and perform some operation based on the information contained in the file?
  • if yes, how do you know which "builder lib" to invoke for each srcs entry, going just by filename is possible I guess but not necessarily very robust?
• deps seem to be doing version resolution, does that mean there needs to be a lockfile as well? Have you considered not doing version resolution, i.e. requiring fixed versions? It seems to me like most dependencies would come from language ecosystems, which usually have their own lockfile already, or is the idea to eventually specify all dependencies through atoms?
• I am not sure I understand the provide block

Not very structured or anything, but I hope it serves as you as some early feedback 👍

[GrizzlT]

• How are srcs used in practice? When building a package with nix, doesn't nix take care of the sources? Not saying I don't like this idea, separating source fetching from the derivation recipe internals is an interesting approach. I only wonder how this might go together with multi-variant packages (although I can see ways to get it workable).
• What is the level of granularity we take on platforms?
  • Is the cpu architecture + kernel combination the minimum level of granularity we take across eka?
  • Or do we allow these strings to be anything, given that a community-maintained "systems" atom is published to deal with quirks that might pop up?
  • Should there be more in these strings or not?
• Is only nix supported? Or are atoms meant to be general-purpose? In which case it would be weird to have the pins.*.args section, I feel like this should be a nix:package specific property instead (i.e. in a different section).
• Is this atom meant to wrap nixpkgs or a local package? It is a bit confusing to mention the file to import nixpkgs with whilst still providing arguments as if nixpkgs should be instantiated... This feels a bit off to me.
• Following the points above, I understand that deps (atoms) are different from srcs, but what are pins used for? I feel like there's redundancy here.

Love the progress here! Hope the feedback is useful.

[nrdxp]

• where is the Nix entrypoint specified? Even if there is a default location, it should be possible to override it

The current entrypoint is a main member of the top-level module (mod.nix) of the atom. We can, perhaps make it configurable, or perhaps it is configured by the type

• does it make sense to have a "type"-specific configuration section?

The build configuration options like platform and provide are type specific, that is the main motivation for "types"

• how do you want to determine that https://example.com/v${major}/${version}/pkg.src.tar.xz is an http endpodint and https://github.com/org/mono a Git endpoint? I guess heuristics are possible, as well as checking for the presence of ref, but I think the more robust approach would be to use git instead of url as key for Git repositories

Makes sense, probably a good idea

• are "builder libs" something built-in, which automatically handle for example Cargo.lock files and perform some operation based on the information contained in the file?

This is part of the motivation for a plugin interface to eka, so one might handle language specific concerns elegantly and with tooling appropriate for the ecosystem

• if yes, how do you know which "builder lib" to invoke for each srcs entry, going just by filename is possible I guess but not necessarily very robust?

In the early days, we will likely simply pass the information off to nix2* libs to parse. Once we get some plugins fleshed out it might get a bit more interesting

• deps seem to be doing version resolution, does that mean there needs to be a lockfile as well? Have you considered not doing version resolution, i.e. requiring fixed versions? It seems to me like most dependencies would come from language ecosystems, which usually have their own lockfile already, or is the idea to eventually specify all dependencies through atoms?

There is already a wip lock format as well, not demonstrated here as it's not quite as pedagogically significant. deps specifically refer to other atoms, so these are always Nix dependencies. We aren't very used to versions in Nix libs, etc. I am trying to change that, essentially 😅

• I am not sure I understand the provide block

Maybe I didn't choose the best name, I considered support as well, but switched to provide since entries of the list are provding a concrete instance of the key name. The key here is to specify all the supported variations possible. For example one might run a eka do --all to build every possible permutation of the support build matrix at once, so that they might all be cached, etc. The idea is to avoid the complex logic required to maintain things like pkgsCross or pkgsStatic etc, etc by simply specifying the supported variations declaratively and statically up front.

I skipped it for brevity, but all the strings here should exist as either atom dependencies or legacy pin dependecies

concretely, in this example ld is an "abstract" package availale in the code, not sure from where yet, but say atom.provides.ld which is "filled in" with the concrete linker depending on what is requested on the cli. From here we can efficiently schedule entire matrices of builds on the backend in a discplined way (i.e. typed via the web API recieving the requests)

• How are srcs used in practice? When building a package with nix, doesn't nix take care of the sources? Not saying I don't like this idea, separating source fetching from the derivation recipe internals is an interesting approach. I only wonder how this might go together with multi-variant packages (although I can see ways to get it workable).

Atoms encourage a disciplined approach to locking all dependencies of various concerns, both other atoms, as well as build sources. Nix still fetches these as FODs at build-time, but they are locked and organized such that automated tooling can more easily make assumptions over the inputs, e.g. trivially generate an SBoM.

Flakes encourage doing the wrong thing, in contrast, since all flake inputs are fetched at eval time, which is inappropriate for build sources, but it is by no means obvious this is the case or there is an alternative. It's also disjointed, if you do konw what an FOD is then you have some dependencies in your lock, and some in your code, and it's not obvious to the newcomer at all why that is.

While legacy sources are common via pins, FODs will still be floating around, but say if you got to a point where all of the package set were atomized, you could have an eval mode that makes FOD totally illegal in your code, enforcing all deps come from the specified lock.

We should aspire, slowly over time, to reach this point and make it the default eval mode, with a fallback when legacy pins are necessary.

• What is the level of granularity we take on platforms?

we stick with doubles for now, since that is what nix does, with the caveat that an infinite number of supported variantions are possible via the provide key

• Is the cpu architecture + kernel combination the minimum level of granularity we take across eka?

anything our build system supports, technically. If a new Nix like system used llvm triples, for example, it should be easy enough to support them here. It just depends on what the underlying evaluator expects.

• Or do we allow these strings to be anything, given that a community-maintained "systems" atom is published to deal with quirks that might pop up?

see previous answer

• Should there be more in these strings or not?

[provides] fills in the gap to specify user specific variations. platform is tied directly to the underlying evaluator, but provides is more abstract for us to create whatever variations we wish to support

• Is only nix supported? Or are atoms meant to be general-purpose? In which case it would be weird to have the pins.*.args section, I feel like this should be a nix:package specific property instead (i.e. in a different section).

atoms were designed, and careful attention taken, to remain abstract so that any "nix-like" evaluator can be supported in the future. In addition, the actual git format on the storage side is even more abstract, potentially being adaptable as a generic distributed code package format as an alternative to centralized registries like a crate registry; out of scope for our purposes here though.

• Is this atom meant to wrap nixpkgs or a local package? It is a bit confusing to mention the file to import nixpkgs with whilst still providing arguments as if nixpkgs should be instantiated... This feels a bit off to me.

I only pointed to a file to show it is possible, if no file is specified the default.nix will be assumed (for nix atoms anyway). The file I point to is the file that recieves the arguments to instantiate nixpkgs, so the arguments are valid. So imagine there is implicit code where import pins.foo.entry pins.foo.args. We might make it so that if args is a list we can pass multiple

To make it more obvious or expressive, I've wondered if instead we might simply allow legacy args to be expressed in nix code, perhaps at atom-path/.legacy/args/my-pin.nix, or something similar.

• Following the points above, I understand that deps (atoms) are different from srcs, but what are pins used for? I feel like there's redundancy here.

No redundancy, very specific purposes, srcs are build sources fetched at build time using FODs. Necessary for the purposes of proper separation and efficiency (deferred to build time). deps are atoms and must be published via the atom format to be discoverable. pins are legacy sources which are not in the atom format, which is an essential input type, as most of the code in the early days will still come from legacy expressions, e.g. flakes and legacy nix code.

If atoms became dominant, pins would become less and less important and generally discouraged, if they can be avoided.

[GrizzlT]

While legacy sources are common via pins, FODs will still be floating around, but say if you got to a point where all of the package set were atomized, you could have an eval mode that makes FOD totally illegal in your code, enforcing all deps come from the specified lock.

We should aspire, slowly over time, to reach this point and make it the default eval mode, with a fallback when legacy pins are necessary.

Do you mean that all the fetch* constructs should eventually disappear in favor of this srcs? Does this mean the FODs in all the code are to be replaced by inputs to the nix files?

anything our build system supports, technically. If a new Nix like system used llvm triples, for example, it should be easy enough to support them here. It just depends on what the underlying evaluator expects.

I think it's also interesting to note that nix technically speaking supports cross-compilation to non-nix environments, as in environments where the nix evaluator is not supported, this probably requires static builds or a bundler of sorts to actually run the build result (e.g. riscv64-linux).

The file I point to is the file that recieves the arguments to instantiate nixpkgs, so the arguments are valid. So imagine there is implicit code where import pins.foo.entry pins.foo.args. We might make it so that if args is a list we can pass multiple

Does it make sense to apply arguments to a pin? Wouldn't it be easier to let the internal package layer instantiate something like nixpkgs and apply arguments there?

The idea is to avoid the complex logic required to maintain things like pkgsCross or pkgsStatic etc, etc by simply specifying the supported variations declaratively and statically up front.

I like this a lot. Especially since it doesn't prohibit dependencies still using internal-attributes that are not readily consumed on their own in some cases.

[nrdxp]

Do you mean that all the fetch* constructs should eventually disappear in favor of this srcs? Does this mean the FODs in all the code are to be replaced by inputs to the nix files?

For the sake of consistently tracking all of our dependencies in a lock and, thus, being able to map operations over them trivially, yes. Of course, there might be some nuance to get there, such as having different FODs for different kinds of sources, but the initial implementation will use nix's own builtin FOD to avoid any external dependencies

Does it make sense to apply arguments to a pin? Wouldn't it be easier to let the internal package layer instantiate something like nixpkgs and apply arguments there?

The import keywork is not callable in atom-nix, so at the very least we have to call it implicitly. We could, of course, leave it to the user to pass args from there I suppose. like

# mod.nix
{
  Pkgs = from.nixpkgs {system = conf.platform.bulid; }; # where `import` is implicitly called on the nixpkgs pin
  myPkg = atom.pkgs.callPackage /* ... */;
}

perhaps that is best?

I like this a lot. Especially since it doesn't prohibit dependencies still using internal-attributes that are not readily consumed on their own in some cases.

The main motivation is caching efficiency, long term, so that the backend can easily keep track of what is supported, what it has already built, what needs to be built, etc etc. Since this is all kept static, it should be parsable even without evaluation. But definitely, simplifying code on the Nix side is another major benefit.

[GrizzlT]

Excellent answers, thank you!

perhaps that is best?

I am not against having a bit of both, avoiding repetition without prohibiting more special cases.

[nrdxp]

so I plan to open an EEP for this, might have to wait til the weekend since I'm fairly swamped, but I've already identified some gaps that need to be filled, so hopefully the next draft will be more refined