NixOS Is the Right Infrastructure for AI Agents

Commissioned, Curated and Published by Russ. Researched and written with AI.

What’s New This Week

Published 23 March 2026. No prior updates.

Changelog

Running a software service on a traditional Linux distro is a solved problem. You’ve got package managers, Dockerfiles, Ansible playbooks, a decade of Stack Overflow answers. The tooling is mature and the failure modes are well-understood.

AI agent infrastructure is not that. It fails in ways that are qualitatively different, and most of the standard tooling does not protect you from those failure modes. NixOS does. Not because “declarative is good” – that’s the least interesting part of the story – but because of three specific properties that matter uniquely to agents: dependency isolation at the CUDA layer, immutability against agent-caused drift, and atomic rollbacks when a model update breaks your stack.

The failure modes are different

Consider the dependency problem first. A typical web service depends on a runtime, a few libraries, maybe a database driver. If something breaks, the version conflict is usually visible and shallow.

An AI agent’s dependency graph looks like this: CUDA version, cuDNN version, PyTorch version, Python version, system libraries like libstdc++ and glibc, model weights format, inference framework. All of these have to align. A single pip install can silently break the CUDA alignment – pip doesn’t know about CUDA, and CUDA doesn’t know about pip. The breakage often doesn’t surface until inference produces garbage output or a cryptic CUDA error.

On Ubuntu, the usual approach is to pin everything in a requirements.txt and hope. In practice, the environment drifts. CI passes on a machine where the right libraries happen to be installed. A new team member can’t reproduce the setup. The GPU driver gets updated and PyTorch stops recognizing the device.

The second failure mode is specific to agents. An agent that can install tools, write files, and execute shell commands can drift its own environment. This is not a hypothetical – it’s the default behaviour of any coding agent running without sandboxing. An agent that runs npm install or pip install or curl | bash leaves the host in a different state than it found it. Over time, the host accumulates changes that nobody made a deliberate decision to make. The declared configuration and the actual system state diverge.

The third failure mode is the multi-machine problem. You’re developing on a Mac, running inference on a Linux server, maybe running a lightweight edge agent on a Raspberry Pi. Getting these environments to converge – and stay converged as dependencies change – is largely a manual and error-prone process on traditional distros.

What NixOS actually does

The Nix package manager’s core property is that every package is a pure function of its inputs. The inputs are hashed, and the output lives at a path in the Nix store that includes that hash: something like /nix/store/abc123-pytorch-2.3.0/. Two different builds of the same package with different dependencies produce different store paths. They can coexist. They cannot silently interfere.

This matters for CUDA because you can specify the entire stack – CUDA toolkit, cuDNN, Python, PyTorch – in a flake.nix and pin them all. The flake lockfile records exact revisions of every input. The stack that works today will work in two years, because nothing in the environment can change without going through Nix. There is no pip install that reaches outside the declared graph.

A NixOS system is itself just a Nix derivation. The entire OS – kernel, services, packages, configuration – is the output of evaluating a set of Nix expressions. When you run nixos-rebuild switch, NixOS builds the new system configuration, makes it the boot default, and switches to it. When you run nixos-rebuild switch --rollback, it switches back to the previous generation. The whole system state rolls back atomically. Not “most of it” – all of it. There are no partial upgrades because the Nix model doesn’t have partial upgrades.

For agents, the immutable base layer is the critical property. The Nix store is read-only at runtime. An agent can create files, run processes, modify its own workspace – but it cannot modify the underlying system packages without going through Nix. The declared environment is the actual environment. Agent actions don’t accumulate silently.

A minimal flake.nix for an AI agent development environment looks something like this:

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-25.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          config.allowUnfree = true;  # needed for CUDA
          config.cudaSupport = true;
        };
      in {
        devShells.default = pkgs.mkShell {
          buildInputs = with pkgs; [
            (python311.withPackages (ps: with ps; [
              torch
              transformers
              accelerate
            ]))
            cudatoolkit
            git
          ];
          shellHook = ''
            echo "Agent dev environment ready"
          '';
        };
      });
}

Enter it with nix develop. You get an exact, reproducible environment with pinned versions of everything. No Docker daemon, no image layers, no docker run -v ceremony. Exit the shell and the environment is gone. Your base system is unchanged.

The agent-specific projects worth knowing

rigup.nix is the most directly relevant project. It’s a Nix module system for packaging AI agent instructions, tools, and configuration together as reproducible artifacts called riglets. Each riglet bundles a SKILL.md (the agent-readable documentation), the tools it needs (Nix packages), and optional config files. Riglets are composable – one can declare dependencies on others, and the module system deduplicates them. The full rig is built by Nix and produces a folder of symlinks plus a RIG.md manifest that the agent reads at startup.

This is a different model from “agent skills live in a folder on the filesystem.” With rigup, the skill is a versioned Nix derivation. You can pin it, roll it back, compose it with other skills, and share it as a flake output. The gap between “the skill I declared” and “the skill that’s running” cannot silently widen.

nixified.ai addresses the model distribution problem. Running an LLM locally typically involves: find the right Python version, create a venv, install the inference framework, deal with CUDA version mismatches, download model weights separately, hope everything aligns. nixified.ai packages AI models as self-contained Nix derivations with all dependencies included. Running a model is a single command:

nix run github:nixified-ai/flake#textgen-nvidia

No Docker, no venv setup, no dependency installation. The model’s full environment is the Nix derivation. nixified.ai uses binary substitution via cachix, so the first run pulls pre-built binaries rather than building from source.

microvm.nix is worth noting for agent sandboxing. When you want a coding agent to run with no access to your personal files – disposable if compromised – microvm.nix makes it straightforward to create ephemeral NixOS VMs on a NixOS host. The VM configuration is declared in Nix. Nothing persists on disk unless explicitly shared with the host. Michael Stapelberg’s February 2026 post on coding agent VMs describes this pattern in detail.

llm-agents.nix from Numtide packages AI coding agents and development tools as Nix packages, updated daily. It gives you claude, codex, aider, and others available via a Nix overlay, pinned and reproducible.

The trade-offs

None of this is free.

Nix’s learning curve is real and steep. The Nix expression language is functional and unlike anything else in the sysadmin toolbox. The mental model – store paths, derivations, closures, the module system – takes time to build. Expect two to four weeks before it feels natural, and expect to read a lot of error messages that don’t make obvious sense at first.

CUDA on NixOS is functional, but the proprietary driver situation is more complex than on Ubuntu. NVIDIA drivers have to be carefully matched to the kernel version, and the NixOS configuration for CUDA differs from the standard Ubuntu setup that most inference framework documentation assumes. You will encounter issues that Ubuntu users do not. The community has solutions for most of them, but you’ll spend time finding those solutions.

The Nix store grows. Every build, every previous system generation accumulates in /nix/store. nix-collect-garbage -d removes old generations and unused store paths, but you need to run it. On a busy build machine this becomes a real storage management concern.

The documentation for AI-specific NixOS problems is thinner than Ubuntu’s. The community is active and the discourse forums are helpful, but when you hit a problem with a specific PyTorch version and a specific NVIDIA driver on NixOS, the probability of finding an existing answer is lower than for the equivalent Ubuntu setup. That gap is narrowing, but it’s not gone.

Who should switch now, and who should wait

NixOS makes sense now if you’re running multiple agents across different machines and environment drift is already a real operational cost. If you’re spending time debugging “it worked on the dev box” failures, or if a model update has broken your production stack and you’re manually rolling things back, the learning curve will pay off quickly.

It makes sense if you’re building infrastructure that other people will run and you want the agent’s environment to be a first-class versioned artifact. With Nix, the environment is the code. It can be reviewed, tested, and rolled back with the same tools you use for everything else.

Wait if you’re running a single machine, you’re comfortable with Docker, and environment drift isn’t a problem you’re actually experiencing. NixOS is solving real problems, but it’s the wrong answer if you don’t have those problems yet.

Wait if your team has no Nix experience and the current moment calls for moving fast. The investment in learning Nix is front-loaded. You pay it before you get the benefits.

A post published on birkey.co last week makes the point concretely. The author describes building a voice-to-text agent in Rust, where the coding agent used nix develop to pull in the entire Rust toolchain, compile the project in an isolated shell, and produce a working binary – without touching the host system. No ~/.cargo, no ~/.rustup, no PATH mutations left behind. Without Nix, the agent would have reached for curl | sh to install rustup and quietly mutated the environment. With Nix, the experiment was clean, reproducible, and committable as a flake.nix.

That’s the core of it. Not that declarative config is philosophically superior, but that the specific failure modes of AI agent infrastructure – CUDA hell, environment drift caused by the agents themselves, multi-machine divergence – have a structural solution in NixOS that doesn’t exist anywhere else in the current tooling landscape. Whether the complexity is worth it depends entirely on whether those failure modes are costing you time now.