Support for SwiftPM (dependencies) #113
Description
So we've been batting around ideas about this since forever, but I think it's time we moved this into public view, to make sure anyone affected has a heads up and can weigh in if interested. I volunteer as tribute.
Background
We need to formulate a proper integration plan for SPM packages. We should support SPM packages as well as we possibly can, and in particular we need to support the case of using SPM packages as AT build dependencies, so that AT users have access to the whole ecosystem of software.
Supporting the opposite way (AT packages as dependencies to an SPM user) is obviously an important goal to all of us, especially so as to not fragment the ecosystem which is undesirable for a wide variety of reasons. However it is technically tricky. We are a superset of SPM packaging format and do all manner of things they don't support and that cannot be represented in their format. For a few examples, we support external scripts, we have unlimited tasks per package, we have umbrella headers, per-package compile flags, plugins, binary packages (soon™), bitcode, iOS support, many more. Many of our packages actually need this stuff to even build, which is what motivated our work in the first place. So it is not a matter of slimming down our featureset or pushing the problem out to individual libraries to somehow write a Package.swift that cannot actually be written.
Meanwhile, SPM doesn't have any way currently to ingest a binary into its buildchain like our atbin system, so we can't simply hand SPM something from our toolchain and get SPM to do a link. I raised this issue very early on and build-dev had a discussion about supporting/allowing external build systems and/or external build products but the response was mixed and AFAIK the discussion has never been resumed. Based on that thread and the subsequent non-revisiting of the topic, it seems unlikely to me that anything here will materialize in SPM for Swift 3, if it materializes ever.
As these individual decisions were more or less intentional on all sides, there may be no sane way to handle SPM->AT dependency graph (there may be insane ways). But anyway that is a harder problem than merely AT->SPM, which is straightforward, entirely within our control, and should be tackled first.
Choices
Broadly speaking, we have two paths to allowing SPM dependencies I'm aware of:
- Natively work with SPM packaging in atpm and/or atbuild, possibly at the atpkg level since it's common functionality. Behind this door our tools are taught about SPM packaging format, and possibly do conversions internally.
- Develop a separate conversion tool that ingests
Package.swiftand spits outbuild.atpkg. You could theoretically run this program on the CLI, and/or various AT tools could try to run it automatically.
The second is cleaner, more in like with AT/Unix philosophy, and provides a nicer integration for several reasons. Obviously it helps folks who want to straight-out migrate an existing package, but it also provides an easy way to support both systems (many of us are in this case) or who just want to see what our syntax is for some case that is more familiar to them in Package.swift syntax.
The first, however, has one key UX advantage. If we want atpm fetch to be able to actually recursively fetch all dependencies, then we may have no choice other than to teach atpm about SPM in a very fundamental way. (Although arguably, we could have it scan for Package.swift and then run the conversion tool if installed as a runtime dependency.)
Moving forward
We have some more time to noodle around on how to design this, but I do think it is important to have best-effort compatibility in time for Swift 3.0. This will require multi-project and multi-maintainer coordination, so I want to make sure we have plenty of time to reach a conclusion on design and get it in.
Composeability is in our charter and while composing with SPM presents some unique challenges, it is just as important as the composition we do already do with AT-family tools (atbuild/atpm/xcode-emit/etc.) and the non-AT family tools (make, Caroline, etc.) Our values are to use the best tool for the job, and we want to be as frictionless as we can for users to choose that tool, whether or not it's a tool we ship.
Finally, it is important to remember that everyone who is an AT contributor as well as everyone in the broader Swift community all want Swift to win. There are some very visible technical disagreements on how exactly to get there. But those differences should never overshadow the big picture that we all want the same outcome, and getting that outcome requires working closely together, or at least as closely as our substantive differences will allow.