devenv is switching Nix implementation to Tvix
In February 2020, I went on a 16-day, 1200km moped trip in northern Thailand with two friends.
As we drove mopeds multiple hours per day, I was listening to an audiobook crossing the chasm. The book goes in depth how technology is challenged to reach mainstream adoption.
The years that followed, I pondered about the disconnect between Nix's devoted user base and its lack of widespread adoption in the broader tech community.
Over 2021 and 2022, I've focused efforts and started nix.dev, writing tutorials and at the end realizing that purely improving documentation can only get us that far.
We needed to fix the foundations.
The negative feedback loop
Thus over the years at Cachix we've talked to each team that abandoned Nix and observed that the pattern was surprisingly consistent.
Nix was introduced by someone being enthusiastic about the technology, but due to a steep adoption curve it was later abandoned due to backlash in the team.
Making it trivial for a project to adopt and maintain a development environment is crucial for other team members to see the benefits of Nix.
For example Shopify was vocal about Nix in 2020, but after that it went quiet. Having companies like Shopify adopt Nix would be a major step forward for the whole ecosystem.
Interface as the heart of Developer Experience
Since 0.1 release two years ago, we've been iterating on declarative interface for developer environments, featuring over 50 languages and over 30 services:
{ pkgs, config, ... }: {
packages = [
pkgs.cargo-watch
];
languages.rust = {
enable = true;
channel = "nightly";
rustflags = "-Z threads=8";
targets = [ "wasm32-unknown-unknown" ];
};
processes = {
backend.exec = "cargo watch -x run";
};
services = {
postgresql.enable = true;
};
}
Since the introduction of tasks in 1.2 release and
Nix caching in 1.3, we're pretty happy with
the devenv command line interface and extensible nature of the module system.
The modular architecture of the module system allows for seamless addition, modification, and removal of configuration options. This flexibility extends to defining your own options for the software you're writing.
Why do we need a Nix rewrite?
We've been using Nix command line interface as a low-level API under the hood to implement interaction with the evaluator and Nix store, although from the first day we wanted to be using an SDK instead. However, the command line interface was the most sensible interface two years ago given the available options.
While utilizing C FFI (Foreign Function Interface) could potentially provide a viable solution, it would necessitate substantial development effort and still leave us vulnerable to memory-safety issues. Moreover, the architecture of the Nix codebase is structured more as a monolithic framework rather than a modular library, which contrasts with Tvix's design philosophy (hence the name).
Fortunately Florian Klink, tazjin and others started writing Tvix in 2021.
Tvix, reimplemented in Rust, offers memory safety and a library-oriented architecture with independently usable components. Leveraging Rust's abstractions and ecosystem (e.g., tracing.rs) will significantly enhance our development process.
There are many architectural differences besides just the obvious Rewrite In Rust, so we'll talk about them as we start replacing command line interface for Tvix, starting with the evaluator.
Integrating Tvix evaluator
The Nix evaluator directly traverses the abstract syntax tree (AST) during evaluation, while Tvix uses a bytecode virtual machine crafted according to crafting interpreters book.
Tvix compiles Nix code into compact bytecode, then executes it in a virtual machine. This two-step approach offers potential performance benefits and optimization opportunities, like many other interpreted languages.
In order to be able to integrate the evaluator we need to:
- Finish implementing builtins.fetchTree, where we have some ideas how to improve the caching layer and get rid of annoying api.github.com dependency that causes rate limiting.
- Implement evaluation debugger, that will allow inspecting program's state in case of errors.
- Finish implementing tvix-eval-jobs, that's going to be used for regression tests against nixpkgs to make sure evaluator behaves correctly.
- Debugging tooling for when we discover regressions in the evaluator.
- Integrate nix-daemon layer to communicate scheduling builds.
We've also recently streamed let's explore Tvix evaluator for those interested digging into the code.
Using language's package manager as the build system
Once we have the evaluator integrated, we can finally generalize building software using Nix reproducible builds, by running underlying build system to generate Nix expressions:
graph TD
A[devenv];
A -->|Rust| C[Cargo];
A -->|JavaScript| D[npm];
A -->|PHP| E[Composer];
C -->|Cargo.lock| F{Nix};
D -->|package.json| F{Nix};
E -->|composer.lock| F{Nix};In Build Systems à la Carte, Nix is labelled as a suspending task scheduler.
In general case the dependency graph is computed statically, but a dependency can declare its dependencies dynamically as part of the build by returning more Nix code.
That's when evaluation and building phases start to mix, with evaluation depending on the result of a build, which is typically called import from derivation (as the naming comes from the implementation).
sequenceDiagram
autonumber
participant NixEvaluator as Nix evaluator
participant NixStore as Nix store
NixEvaluator->>NixEvaluator: evaluate
NixEvaluator->>NixStore: write derivation
NixStore->>NixStore: build
NixStore->>NixEvaluator: read derivation output
NixEvaluator->>NixEvaluator: evaluateWith single-threaded evaluation, the process described above gets blocked on each build requested during evaluation.
Implementing parallel evaluation in Tvix, made possible by Rust's robust concurrency primitives, will unlock the ability to support automatic conversion of language-specific build systems into Nix without sacrificing developer experience.
Final Words
As we embark on this new chapter with Tvix, I'm reminded of the journey that brought us here. It's been a decade since I wrote we can do better blog post, highlighting the potential for improvement in configuration management and development environments and I'm glad it's all coming together.
Keep an eye out for updates and discussions:
- Attend "devenv is switching to Tvix" talk at NixCon to learn more about our plans and progress.
- If you'd like to help out hacking on Rust and Tvix, join us at OceanSprint for a week of hacking.
- Join devenv Discord community.
- Subscribe to the mailing list for future announcements at the bottom of the page.
Domen
