-
Notifications
You must be signed in to change notification settings - Fork 50
Reproducible development environment and preparation for publishing NGS into nix package manager to increase adoption #693
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| use flake | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -21,3 +21,6 @@ pcre_constants.include | |
| junk | ||
|
|
||
| .vagrant/ | ||
|
|
||
| .direnv/ | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Direnv creates .direnv to maintain its state. |
||
| result | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nix creates result/ directory when you run |
||
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| { | ||
| description = "NGS - Next Generation Shell"; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's the name of the show! |
||
|
|
||
| inputs = { | ||
| nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This looks scarier than it is. All the cool kids are on unstable. I used to use Arch, btw |
||
| flake-utils.url = "github:numtide/flake-utils"; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nix build system didn't "find" flake framework of unification of inputs / outputs for Nix expression straight away. Before we used to write simple nix programs which would be [lazily] evaluated top to bottom. flake-utils is like flake framework stdlib. It has some convenience functions like generating an attrset per each system supported by default (used here), to flatten a bunch of nested attributes into a flat attrset , etc. |
||
| }; | ||
|
|
||
| outputs = | ||
| { | ||
| self, | ||
| nixpkgs, | ||
| flake-utils, | ||
| }: | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is a function saying "give me a reference to the outputs of this function (recursive), everything that is packaged in whatever the source of nix packages is (unstable "channel" of official nix packages in this case) and flake utils, and I will give you an attrset defined below". |
||
| flake-utils.lib.eachDefaultSystem ( | ||
| system: | ||
| let | ||
| pkgs = nixpkgs.legacyPackages.${system}; | ||
|
|
||
| # Local source filter - exclude build artifacts | ||
| localSrc = builtins.path { | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Source code is very important to build software! We only care about the actual source tree, not build artefacts or git object store. |
||
| path = ./.; | ||
| name = "ngs-source"; | ||
| filter = | ||
| path: type: | ||
| let | ||
| baseName = baseNameOf path; | ||
| in | ||
| !(baseName == "build" || baseName == "result" || baseName == ".git" || baseName == ".direnv"); | ||
| }; | ||
|
|
||
| # For local development, override the package to use local source | ||
| ngs = (pkgs.callPackage ./package.nix { }).overrideAttrs (oldAttrs: { | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. When we buuld ngs for nixpkgs (this is my end goal with this work -- to have ngs included into official nixpkgs unstable), we want to pin the version to a certain place fetchable with some fetcher. We use github fetcher, because github fetcher is so easy to use. In package.nix you will see that src is remote! But to build stuff locally, we need to override this source with local source. |
||
| src = localSrc; | ||
| version = oldAttrs.version + "-dev"; | ||
| }); | ||
| in | ||
| { | ||
| packages = { | ||
| default = ngs; | ||
| ngs = ngs; | ||
| }; | ||
|
|
||
| devShells.default = pkgs.mkShell { | ||
| inputsFrom = [ ngs ]; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is where the package is coming from, so we just reuse release dependencies from there. We separate concerns and oly add developmnent dependencies here like valgrind profiler, gdb, and other tools I'm scared to launch. |
||
|
|
||
| packages = | ||
| with pkgs; | ||
| [ | ||
| # Development tools | ||
| gdb | ||
| clang-tools # clangd, clang-format | ||
| ] | ||
| ++ lib.optionals stdenv.isLinux [ | ||
| valgrind | ||
| strace | ||
| ]; | ||
|
|
||
| shellHook = '' | ||
| echo "╔══════════════════════════════════════════════════════════╗" | ||
| echo "║ NGS Development Environment ║" | ||
| echo "╠══════════════════════════════════════════════════════════╣" | ||
| echo "║ Build: cmake -B build && cmake --build build ║" | ||
| echo "║ Test: cd build && ctest --output-on-failure ║" | ||
| echo "║ Run: NGS_PATH=lib ./build/ngs ║" | ||
| echo "║ Package: nix build ║" | ||
| echo "╚══════════════════════════════════════════════════════════╝" | ||
| ''; | ||
| }; | ||
|
|
||
| # For nix fmt | ||
| formatter = pkgs.nixpkgs-fmt; | ||
| } | ||
| ); | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| { | ||
| lib, | ||
| stdenv, | ||
| fetchFromGitHub, | ||
| fetchurl, | ||
| cmake, | ||
| pkg-config, | ||
| pandoc, | ||
| gawk, | ||
| makeWrapper, | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess I need to emphasise this one! When CMake installs your software, it assumes LFS compliance. With nix everything lives in /nix/store, thus at runtime software built with Nix often can't find libraries. makeWrapper replaces global runtime assumptions with explicit, reproducible runtime configuration. But one picture is better than 1000 words: |
||
| boehmgc, | ||
| json_c, | ||
| libffi, | ||
| pcre, | ||
| # Darwin-specific: GNU sed is required for build scripts | ||
| gnused, | ||
| }: | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think this is understandable, it just follows the list of dependencies that you have listed in your scripts. |
||
|
|
||
| stdenv.mkDerivation (finalAttrs: { | ||
| pname = "ngs"; | ||
| version = "0.2.17"; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Apr 5th release |
||
|
|
||
| src = fetchFromGitHub { | ||
| owner = "ngs-lang"; | ||
| repo = "ngs"; | ||
| rev = "v${finalAttrs.version}"; | ||
| hash = "sha256-j7OAXHADc2LlabKxVgYiKeDDtLttDVIavhQZSGyPGlE="; | ||
| }; | ||
|
|
||
| # NGS requires peg 0.1.18 specifically due to custom patches in build-scripts/ | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That was a bit of a pain in the rear to package, but I managed. I'm lucky that it seems that everything is mutually compatible between nixpkgs-unstable and peg-0.1.18! |
||
| # The build scripts patch the leg output for location tracking, and this | ||
| # patching is not compatible with newer peg versions (0.1.20+). | ||
| # CMake will download and build peg 0.1.18 automatically when leg is not found. | ||
| pegSrc = fetchurl { | ||
| url = "https://www.piumarta.com/software/peg/peg-0.1.18.tar.gz"; | ||
| hash = "sha256-IBk73Wc/x0h6OJN+KX//CKpzdRtjOghqwow7NIkPkIQ="; | ||
| }; | ||
|
|
||
| nativeBuildInputs = | ||
| [ | ||
| cmake | ||
| pkg-config | ||
| pandoc | ||
| gawk | ||
| makeWrapper | ||
| ] | ||
| ++ lib.optionals stdenv.isDarwin [ | ||
| gnused | ||
| ]; | ||
|
|
||
| buildInputs = [ | ||
| boehmgc | ||
| json_c | ||
| libffi | ||
| pcre | ||
| ]; | ||
|
|
||
| # The build scripts require GNU sed; on Darwin we need to ensure it's found | ||
| # Also pre-populate the peg source to avoid network access during build | ||
| preConfigure = | ||
| '' | ||
| # Create the external project download directory | ||
| mkdir -p build/leg-prefix/src | ||
| # Copy peg source to where CMake ExternalProject expects it | ||
| cp ${finalAttrs.pegSrc} build/leg-prefix/src/peg-0.1.18.tar.gz | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Here we are tricking CMake into thinking that it already downloaded the peg code by placing it where it expects. The reason we have to maintain our patches ourselves is that nix is hermetic and side-effect-free at the build time, meaning it can't just download stuff from the internet willy nilly. |
||
| '' | ||
| + lib.optionalString stdenv.isDarwin '' | ||
| export PATH="${gnused}/bin:$PATH" | ||
| ''; | ||
|
|
||
| cmakeFlags = [ | ||
| "-DBUILD_MAN=ON" | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Pure cargo cult
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. (as in, I didn't check that it actually builds manual)
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. mmm. We don't expect people to
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, what I mean is that I just copied this flag without checking it. I don't even know if it's real. I do expect people to but I don't know if the man pages are generated because of this flag and I don't know if they are populated correctly. :) |
||
| ]; | ||
|
|
||
| # Set NGS_PATH so the installed binary can find the standard library | ||
| postInstall = '' | ||
| wrapProgram $out/bin/ngs \ | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. wrapProgram is coming from makeWrapper package and this is what makes the wrapper that I showed you before. |
||
| --set NGS_PATH $out/lib/ngs | ||
| ''; | ||
|
|
||
| meta = with lib; { | ||
| description = "Next Generation Shell - a powerful programming language and shell designed for Ops"; | ||
| longDescription = '' | ||
| NGS is a unique combination of select features borrowed from other | ||
| languages and original features. NGS was built from the ground up | ||
| focusing on daily systems engineering tasks. | ||
| One way to think about NGS is bash plus data structures plus better | ||
| syntax and error handling. Scripting AWS is much easier with NGS, | ||
| there is a Declarative Primitives style library for that. | ||
| ''; | ||
| homepage = "https://ngs-lang.org/"; | ||
| changelog = "https://github.com/ngs-lang/ngs/blob/v${finalAttrs.version}/CHANGELOG.md"; | ||
| license = licenses.gpl3Only; | ||
| maintainers = with maintainers; [ ]; | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What do I do when something breaks and I don't have enough understanding how to fix this?
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm afraid I will have to be maintainer then, the issue is that I'm currently using YSH and I wanted to evaluate NGS as a better alternative. There is also a potential that someone in a huge nix community wants to take up maintenance of this expression. We will see!
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I see you like the idea largely. I will put myself as a maintainer then and will remove myself if I feel like I can't do maintenance of it anymore or when I find someone else trustworthy to maintain this set of exprs. |
||
| mainProgram = "ngs"; | ||
| platforms = platforms.unix; | ||
| }; | ||
| }) | ||

There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This change is there purely for ergonomics of people who are using
direnv-nix."use flake" directive means "dear direnv, when the user enters this directory, find file flake.nix and evaluate it with
nix develop".What happens when you run
nix developis that nix installs all the necessary packages into/nix/storebehind their input hashes and then "activates" a mutually-compatible ser of packages, giving access to them in PATH.