Reproducible development environment and preparation for publishing NGS into nix package manager to increase adoption #693

cognivore · 2025-11-29T18:11:36Z

Why

NGS deserves better packaging. Right now if someone wants to try NGS on NixOS or with nix, they're out of luck — no nix-shell -p ngs, no declarative system config, nothing. This is a shame because NGS is exactly the kind of tool that nix users would appreciate: a shell that actually learns from PLT proceedings beyond 1985 ❤️

What

Standard cmake-based package with a few quirks handled:

peg version pinning: NGS patches the leg parser output in build-scripts/patch-leg-output.sed to add location tracking. These patches assume peg 0.1.18's output format.

Nixpkgs ships 0.1.20 which generates slightly different code (extra yySet() calls that don't get patched), causing build failures.

Rather than patching NGS's sed scripts and diverging from upstream, we pre-fetch peg 0.1.18 and let CMake's ExternalProject use it — same as the upstream build does when leg isn't found.

Darwin support: Build scripts use GNU sed extensions. Added gnused to nativeBuildInputs on Darwin and prepend to PATH.

NGS_PATH: Binary wrapped to find stdlib at $out/lib/ngs.

Man pages: Enabled via -DBUILD_MAN=ON, requires pandoc.

Testing

Built and tested on darwin-aarch64 (only!):

$ nix build && ./result/bin/ngs -p 'sum(1..100)'
4950

$ ./result/bin/ngs -p '[1,2,3].map(X*2)'  
[2,4,6]

$ ./result/bin/ngs -p '{"a": 1}.mapv(X+10)'
{a=11}

$ ls result/share/man/man1/
na.1.gz  ngs.1.gz  ngsint.1.gz  ngslang.1.gz  ngsstyle.1.gz  ngstut.1.gz  ngswhy.1.gz

That's it. If you are happy with it, I will submit the package to pkgs/by-name/ng/ngs/package.nix so that Nix users can easily benefit from NGS.

Solution: - Add nixpkgs-compatible package.nix for upstream contribution: - Use stdenv.mkDerivation with finalAttrs pattern (package.nix) - Fetch source from GitHub with verified hash (package.nix) - Pre-fetch peg 0.1.18 to avoid network during build (package.nix) - Handle Darwin/Linux differences via lib.optionals (package.nix) - Wrap binary with NGS_PATH for stdlib discovery (package.nix) - Include comprehensive meta with GPL-3.0 license (package.nix) - Add flake.nix for local development workflow: - Override package.nix to use local source with -dev suffix (flake.nix) - Filter build artifacts from source tree (flake.nix) - Provide devShell with cmake, clang-tools, gdb (flake.nix) - Support all four platforms via flake-utils (flake.nix) - Add direnv integration for seamless development (.envrc) - Pin flake inputs for reproducibility (flake.lock) Technical notes: - peg version constraint: NGS build-scripts/patch-leg-output.sed patches the leg parser generator output to add location tracking. These patches modify function signatures (yyDo, yyPush, yyPop, yySet) to accept a 4th location[4] parameter. Nixpkgs ships peg 0.1.20 which generates code with direct yySet() calls that the sed script doesn't patch, causing compilation errors. Solution: pre-fetch peg 0.1.18 tarball and populate CMake's ExternalProject download directory in preConfigure. - NGS_PATH discovery: The NGS binary searches for stdlib.ngs in NGS_PATH. Without wrapping, it would fail to find the standard library. The postInstall phase uses wrapProgram to set NGS_PATH=$out/lib/ngs. - Darwin compatibility: Build scripts use GNU sed extensions (like \+). On macOS, /usr/bin/sed is BSD sed. Solution: add gnused to nativeBuildInputs on Darwin and prepend to PATH in preConfigure. Verified working: $ nix build && ./result/bin/ngs -p 'sum(1..100)' 4950 $ ./result/bin/ngs -p '[1,2,3,4,5].map(X*2)' [2,4,6,8,10] $ ./result/bin/ngs -p '{"a": 1, "b": 2}.mapv(X+10)' {a=11, b=12} $ ls ./result/share/man/man1/ na.1.gz ngs.1.gz ngsint.1.gz ngslang.1.gz ngsstyle.1.gz ngstut.1.gz ngswhy.1.gz

Solution: - Use default formatter on Nix files to comply with follow-up changes by other Nix enthusiasts

ilyash-b · 2025-11-30T08:54:32Z

Sounds good. Thanks! I'll need to review it. I plan to do it in the next few weeks.

ilyash-b · 2025-12-13T06:05:32Z

Started looking. As a person with no Nix experience I'm not fast at this to put it mildly. Any tips to accelerate my understanding of the PR (changed files) are appreciated.

cognivore · 2025-12-13T15:04:14Z

.envrc

This change is there purely for ergonomics of people who are using direnv-nix.

"use flake" directive means "dear direnv, when the user enters this directory, find file flake.nix and evaluate it with nix develop".

What happens when you run nix develop is that nix installs all the necessary packages into /nix/store behind their input hashes and then "activates" a mutually-compatible ser of packages, giving access to them in PATH.

cognivore · 2025-12-13T15:04:36Z

.gitignore

 .vagrant/
+
+.direnv/
+result


Nix creates result/ directory when you run nix build. Symlinks to built artefacts in /nix/store live there

cognivore · 2025-12-13T15:04:46Z

.gitignore


 .vagrant/
+
+.direnv/


Direnv creates .direnv to maintain its state.

cognivore · 2025-12-13T15:04:55Z

flake.nix

@@ -0,0 +1,75 @@
+{
+  description = "NGS - Next Generation Shell";


That's the name of the show!

cognivore · 2025-12-13T15:05:16Z

flake.nix

+  description = "NGS - Next Generation Shell";
+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";


This looks scarier than it is. All the cool kids are on unstable.

I used to use Arch, btw

cognivore · 2025-12-13T15:10:44Z

flake.nix

+
+  inputs = {
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+    flake-utils.url = "github:numtide/flake-utils";


Nix build system didn't "find" flake framework of unification of inputs / outputs for Nix expression straight away. Before we used to write simple nix programs which would be [lazily] evaluated top to bottom.

flake-utils is like flake framework stdlib.

It has some convenience functions like generating an attrset per each system supported by default (used here), to flatten a bunch of nested attributes into a flat attrset , etc.

cognivore · 2025-12-13T15:12:57Z

flake.nix

+      self,
+      nixpkgs,
+      flake-utils,
+    }:


This is a function saying "give me a reference to the outputs of this function (recursive), everything that is packaged in whatever the source of nix packages is (unstable "channel" of official nix packages in this case) and flake utils, and I will give you an attrset defined below".

cognivore · 2025-12-13T15:14:25Z

flake.nix

+        pkgs = nixpkgs.legacyPackages.${system};
+
+        # Local source filter - exclude build artifacts
+        localSrc = builtins.path {


Source code is very important to build software!

We only care about the actual source tree, not build artefacts or git object store.

cognivore · 2025-12-13T15:17:25Z

flake.nix

+        };
+
+        # For local development, override the package to use local source
+        ngs = (pkgs.callPackage ./package.nix { }).overrideAttrs (oldAttrs: {


When we buuld ngs for nixpkgs (this is my end goal with this work -- to have ngs included into official nixpkgs unstable), we want to pin the version to a certain place fetchable with some fetcher. We use github fetcher, because github fetcher is so easy to use.

In package.nix you will see that src is remote!

But to build stuff locally, we need to override this source with local source.

cognivore · 2025-12-13T15:20:31Z

flake.nix

+        };
+
+        devShells.default = pkgs.mkShell {
+          inputsFrom = [ ngs ];


This is where the package is coming from, so we just reuse release dependencies from there. We separate concerns and oly add developmnent dependencies here like valgrind profiler, gdb, and other tools I'm scared to launch.

cognivore · 2025-12-13T15:23:24Z

package.nix

+  pcre,
+  # Darwin-specific: GNU sed is required for build scripts
+  gnused,
+}:


I think this is understandable, it just follows the list of dependencies that you have listed in your scripts.

cognivore · 2025-12-13T15:23:48Z

package.nix

+
+stdenv.mkDerivation (finalAttrs: {
+  pname = "ngs";
+  version = "0.2.17";


Apr 5th release

cognivore · 2025-12-13T15:24:50Z

package.nix

+    hash = "sha256-j7OAXHADc2LlabKxVgYiKeDDtLttDVIavhQZSGyPGlE=";
+  };
+
+  # NGS requires peg 0.1.18 specifically due to custom patches in build-scripts/


That was a bit of a pain in the rear to package, but I managed.

I'm lucky that it seems that everything is mutually compatible between nixpkgs-unstable and peg-0.1.18!

cognivore · 2025-12-13T15:29:33Z

package.nix

+  pkg-config,
+  pandoc,
+  gawk,
+  makeWrapper,


I guess I need to emphasise this one!

When CMake installs your software, it assumes LFS compliance.

With nix everything lives in /nix/store, thus at runtime software built with Nix often can't find libraries.

makeWrapper replaces global runtime assumptions with explicit, reproducible runtime configuration.

But one picture is better than 1000 words:

sweater in pentavus~/Github/ngs on  master via C v21.1.2-clang via △ v4.1.2 via 𝗩 via  impure (nix-shell-env) on  (us-east-1) λ nix build sweater in pentavus~/Github/ngs on  master via C v21.1.2-clang via △ v4.1.2 via 𝗩 via  impure (nix-shell-env) on  (us-east-1) λ cat result/bin/ngs #! /nix/store/p0k9r5h8qs7220xdbdihhfgzwjcly70x-bash-5.3p3/bin/bash -e export NGS_PATH='/nix/store/lhlnv03mjklkc8dwc2gvmxs6axwshbzg-ngs-0.2.17-dev/lib/ngs' exec -a "$0" "/nix/store/lhlnv03mjklkc8dwc2gvmxs6axwshbzg-ngs-0.2.17-dev/bin/.ngs-wrapped" "$@" sweater in pentavus~/Github/ngs on  master via C v21.1.2-clang via △ v4.1.2 via 𝗩 via  impure (nix-shell-env) on  (us-east-1) λ head -n1 result/bin/.ngs-wrapped �� H__PAGEZERO��__TEXT@@__text__TEXTD0(uD0�__stubs__TEXTl��l� __const__TEXT �T �__cstring__TEXTt��t�__unwind_info__TEXT$9�$9��__DATA_CONST@@@@�__got__DATA_CONST@�@��__DATA��__data__DATA��S�__common__DATA��

cognivore · 2025-12-13T15:29:57Z

package.nix

+    '';
+
+  cmakeFlags = [
+    "-DBUILD_MAN=ON"


Pure cargo cult

(as in, I didn't check that it actually builds manual)

mmm. We don't expect people to man ngs?

No, what I mean is that I just copied this flag without checking it. I don't even know if it's real. I do expect people to man ngs, I plan to even do it myself. Just flagging that I didn't check that it actually works, I just did this:

$ ls result/share/man/man1/ na.1.gz ngs.1.gz ngsint.1.gz ngslang.1.gz ngsstyle.1.gz ngstut.1.gz ngswhy.1.gz

but I don't know if the man pages are generated because of this flag and I don't know if they are populated correctly. :)

cognivore · 2025-12-13T15:31:23Z

package.nix

+
+  # Set NGS_PATH so the installed binary can find the standard library
+  postInstall = ''
+    wrapProgram $out/bin/ngs \


wrapProgram is coming from makeWrapper package and this is what makes the wrapper that I showed you before.

sweater in pentavus~/Github/ngs on  master via C v21.1.2-clang via △ v4.1.2 via 𝗩 via  impure (nix-shell-env) on  (us-east-1) λ cat result/bin/ngs #! /nix/store/p0k9r5h8qs7220xdbdihhfgzwjcly70x-bash-5.3p3/bin/bash -e export NGS_PATH='/nix/store/lhlnv03mjklkc8dwc2gvmxs6axwshbzg-ngs-0.2.17-dev/lib/ngs' exec -a "$0" "/nix/store/lhlnv03mjklkc8dwc2gvmxs6axwshbzg-ngs-0.2.17-dev/bin/.ngs-wrapped" "$@"

cognivore · 2025-12-13T15:31:56Z

package.nix

+    homepage = "https://ngs-lang.org/";
+    changelog = "https://github.com/ngs-lang/ngs/blob/v${finalAttrs.version}/CHANGELOG.md";
+    license = licenses.gpl3Only;
+    maintainers = with maintainers; [ ];


What do I do when something breaks and I don't have enough understanding how to fix this?

I'm afraid I will have to be maintainer then, the issue is that I'm currently using YSH and I wanted to evaluate NGS as a better alternative. There is also a potential that someone in a huge nix community wants to take up maintenance of this expression.

We will see!

I see you like the idea largely. I will put myself as a maintainer then and will remove myself if I feel like I can't do maintenance of it anymore or when I find someone else trustworthy to maintain this set of exprs.

cognivore · 2025-12-13T15:37:32Z

package.nix

+      mkdir -p build/leg-prefix/src
+
+      # Copy peg source to where CMake ExternalProject expects it
+      cp ${finalAttrs.pegSrc} build/leg-prefix/src/peg-0.1.18.tar.gz


Here we are tricking CMake into thinking that it already downloaded the peg code by placing it where it expects.

The reason we have to maintain our patches ourselves is that nix is hermetic and side-effect-free at the build time, meaning it can't just download stuff from the internet willy nilly.

cognivore · 2025-12-13T15:39:46Z

Started looking. As a person with no Nix experience I'm not fast at this to put it mildly. Any tips to accelerate my understanding of the PR (changed files) are appreciated.

thank you so much, I have written plain English comments to most of things that I know can trip up people who have never seen nix.

https://github.com/ngs-lang/ngs/pull/693/changes

The only thing my texts assume is that you understand the fundamental principle of Nix:

lazy programming language
evaluations are pure
inputs are all the source codes and arguments
inputs are hashed and outputs (deterministic) are placed into a fixed place in /nix/store.

If you have any follow-up questions, don't hesitate to ask!

ilyash-b

Thanks! I do like the idea!

We have builds in .github/workflows/build.yml for other platforms. Can you please add Nix build there? Or is there any alternative way to know when we break it for Nix?
See couple of inline comments

ilyash-b · 2025-12-15T04:47:47Z

package.nix

+    '';
+
+  cmakeFlags = [
+    "-DBUILD_MAN=ON"


mmm. We don't expect people to man ngs?

ilyash-b · 2025-12-15T04:50:02Z

package.nix

+    homepage = "https://ngs-lang.org/";
+    changelog = "https://github.com/ngs-lang/ngs/blob/v${finalAttrs.version}/CHANGELOG.md";
+    license = licenses.gpl3Only;
+    maintainers = with maintainers; [ ];


What do I do when something breaks and I don't have enough understanding how to fix this?

cognivore · 2025-12-15T13:57:08Z

Thanks! I do like the idea!

We have builds in .github/workflows/build.yml for other platforms. Can you please add Nix build there? Or is there any alternative way to know when we break it for Nix?

See couple of inline comments

Understood! I will try to do it this week. I assume I can test it locally with act somehow? 🍡

cognivore added 2 commits November 29, 2025 18:03

Problem: Nix files lack formatting

271ee54

Solution: - Use default formatter on Nix files to comply with follow-up changes by other Nix enthusiasts

cognivore commented Dec 13, 2025

View reviewed changes

.gitignore

.vagrant/

.direnv/

Copy link

Author

cognivore Dec 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Direnv creates .direnv to maintain its state.

ilyash-b reacted with thumbs up emoji

cognivore commented Dec 13, 2025

View reviewed changes

ilyash-b reviewed Dec 15, 2025

View reviewed changes

		@@ -0,0 +1,75 @@
		{
		description = "NGS - Next Generation Shell";

Reproducible development environment and preparation for publishing NGS into nix package manager to increase adoption #693

Are you sure you want to change the base?

Reproducible development environment and preparation for publishing NGS into nix package manager to increase adoption #693

Uh oh!

Conversation

cognivore commented Nov 29, 2025

Why

What

Testing

Uh oh!

ilyash-b commented Nov 30, 2025

Uh oh!

ilyash-b commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cognivore commented Dec 13, 2025

Uh oh!

ilyash-b left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cognivore commented Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

ilyash-b commented Dec 13, 2025 •

edited

Loading