Cobalt's Homepage

Attic as Nix Cache

Cobalt — Fri, 22 Aug 2025 00:00:00 GMT

Attic as Nix Cache

This article will be a report/guide on how to setup attic in a monolithic setup as nix cache with S3 and PostgreSQL.

This setup is used in my homelab to store around 150 GB of cached data in around 5.000.000 chunks of data. There are still some things that can be improved but this setup has held up well to (ab)use in CI jobs and for distributed builds.

Requirements

It is recommended to use a NixOS host for at least attic. The ressource consumption of attic depends on two factors:

Load, how many paths are you planning to push at peak

This host should have at least one core and at least 2-4 GB of RAM. If you plan to have both the PostgreSQL database and attic on the same server, add few GB and a core.

Arch

As the introduction hints at this setup is intended to run attic in monolithic mode, i.e., one atticd service fulfills all internal attic roles. This servers is backed by a PostgreSQL instance for metadata storage and S3 for the actual data storage.

The S3-compatible service choosen here was garage but I've also used minio for the same role in the past. Please be aware that some cloud solutions, like CloudFront or R2, have known compatability issues with attic.

Notes on other configurations

I have previously run a SQLite + local storage setup with attic. This worked fine during testing, but the setup fell apart when handling load from multiple CI jobs (especially for big uploads).

In particular, there were a lot of issues with concurrent write contention on SQLite and (not relevant anymore) some bugs with the storage layer. My recommendation would be to go with at least PostgreSQL and, if available, also stick to S3. The current setup has been mostly solid with some issues during mass uploads, where basic PostgreSQL tuning[^psql] was required to increases performance.

[^psql]: My mass uploads could lead to over 500 QPS, which is not a lot for PostgreSQL but enough to make some tuning worth it. After viewing collected stats, it also turned out that > 95% of queries were cache hits. Based on this, I provided the PostgreSQL more cache memory and saw an increases in cache hits and lower CPU load as a, likely direct, result of the former. I also adjusted some other concurrency related options, based on pgtune, but did not see a significant change in performance based on these.

Guide

Let's start with the host for attic itself, you should

Migrating from MinIO to garage

Cobalt — Fri, 22 Aug 2025 00:00:00 GMT

Migrating to Garage

Background

:::note If you only care about the technical stuff, you can skip to the next section. This will be a short summary on why I migrated. :::

Since a few years my GitLab instances and some other services required S3 for distributed file storage.

At the time, MinIO was the obvious choice as it had solid S3 support and a good Web UI. The data on it has now been migrated over three instances, from a docker container to bare-metal on Debian to the final destination as a NixOS services on my NAS.

However, they recently yanked the access management via the Web UI from the OSS edition and moved it to their commercial offering. This part of the WebUI was the one I used the most and at least for me was enough motivation to look for alternatives.

I do understand the motivation from their side to push people towards their enterprise offering, given how popular the free edition of MinIO has become. However, the prospect of MinIO pulling access management features in the future lower it's long-term reliability a lot for me.

:::tip[Panam Palmer] Assessment, Assembly, Action :::

Assessment

The first step of the migration was assessing:

What consumers are present?
What do the consumers require?

The 1. was relatively simple thanks to existing documentation:

GitLab (pages, distributed cache, artifacts)
Outline (attachements)
Plane (attachements)
Attic (artifacts)

GitLab is relatively easy to manage as all buckets can stay private and only services need to access it. However, both outline and plane expose assets to uses, either by signed URLs or via a proxied route.

Assembley

Tooling

The minio-client, mc, was installed for data migration. For S3 API interations, specifically CORS policy migration, the awscli was installed.

If you have nix installed:

use nix shell nixpkgs#awscli nixpkgs#minio-client
or nix-shell -p awscli minio-client

Reducing scope

Both Plane and Outline were reconfigured to use alternative storage methods. This simplied all other steps as at least plane prefers bucket policies.

Preparing the destination

The garage service was deployed as outlined in the Quick Start Guide. This was relatively quick on my NixOS server for a single-node installation.

Preparing the source

For migrating the data, we need to be able to read it. In this case we want to acquire an access key that can read all MinIO buckets.

A new identity was created for this and a key with the readonly policy template worked fine here. This identity can be removed after the migration.

Action

The data migration steps below likely will mean some downtime for your services. It is not covered directly here, but another step was to coordinate service maintenance with this data migration.

For GitLab this meant:

a maintenance window for a GitLab upgrade was also used to migrate pages data
all runners were paused and drained of jobs before the cache was migrated

The draining of runners was likely not required as they can also work with disjunct caches. It was however the easier step to ensure good cache availability for all jobs.

For Attic the service had a small downtime while data was being migrated.

Configuring the destination

All relevant buckets were recreated based on the quick start guide.

This effectively boiled down to:

garage bucket create <bucket>
garage key create <identity>
# NOTE: down key id/ access key for service
garage bucket allow --read|--write <bucket> --key <identity>
# either --read and/or --write

As bucket/acl policies are not supported this was all steps required for the service identities temselves.

Another key with read and write priviledges for all buckets was also created for the migration. It will be used in the next step and can be removed with garage key delete <key> --yes afterwards.

Migrating data

For interacting with S3 storages, both minio and garage, the minio CLI was choosen. Mainly because its mirror subcommand makes it trivial to fully replicate the contents of a bucket.

The migration key id/access key for minio is required for the next step, an alias for the mc cli can be created:

mc alias set source https://<minio> <key id> <secret key>

For garage, another alias with the priviledged migration key was created:

mc alias set garage https://<garage> <migration key id> <migration secret key>

You can test both aliases with mc ls <alias>. They should both list the buckets for each endpoint.

The next step is migrating the objects.

:::warning During data mirroring the service(s) writing to the bucket should be stopped. Otherwise, you might run into data consistency issues afterwards. :::

You can now mirror each bucket with: mc mirror source/<bucket> garage/<bucket>

Porting CORS policies

Some buckets require CORS policies. Especially when a service uses signed URLs or similiar this needs to be trasferred for functional operation.

For this step the awscli was used. The same credentials as the mc aliases were used here.

First, the old policy was pulled from the source bucket and saved to /tmp/cors:

aws --endpoint https://<minio> s3api get-bucket-policy --bucket <bucket> > /tmp/cors

And then pushed to the new bucket:

aws --endpoint https://<garage> s3api put-bucket-cors --bucket <bucket> --cors-configuration file:///tmp/cors

This has to be done on a per-bucket basis.

GitLab CI & Nix

Cobalt — Wed, 06 Aug 2025 00:00:00 GMT

Introduction

At $work we have been using GitLab CI in combination with NixOS package builds, checks and NixOS tests for up to 2 years now. This article will cover a loose collection of learnings and tips starting from optimizing GitLab runners to effective usage of GitLab caches.

As for scope, this article only concerns itself with self-hosted GitLab CE instances. All covered topics should also work on EE but are untested. It is also assumed that you are using flakes however most guidance should also apply to channel-based workflows.

Runners

The most common GitLab runner is the docker runner. It enables you to run jobs inside an OCI container image.

There are however some limitations when using nix inside a normal, unpriviledged docker runner. Notably, the sandbox is disabled as it require elevated priviledges. This can have unintended side effects, like build time access to the internet.

There are two runner configuration types that have helped me here:

Using the host daemon

You can make the host daemon available to the job inside the runner, with the configuration below. This will also mount the host nix store inside the runner's container.

There are advantages to this approach:

strong caching, all jobs on the runner share the same store. This includes all derivations etc. and also makes the use of, e.g., harmonia, for native caching-sharing between runners possible.
sharing of build locks, if two jobs on the same runner try to build the same derivation it will only be built once.
sandboxing, as the builds are delegated to the host daemon, the builders on the host can use the sandbox of the host daemon.

There are however also disadvantages:

minor compatability issues, the runner is likely only suitable for use with nix images. Other images often rely on specific $PATH or other variables that may be altered in unexpected ways. The trivial workaround for this is to disable running of untagged jobs.
runner overload, as all builds are on the host no direct CPU or memory limits can be enforced on the builds of any single job. This is especially a problem when multiple heavy jobs (NixOS tests) get issued to the same runner.
priviledged access, the jobs will likely need to connected as trusted-users. This means they effectively have control over the host daemon and should trusted as priviledge escalation is significantly easier.

Based on the NixOS wiki article on GitLab runners, below is an adjusted configuration:

{ pkgs, lib, config, ... }: {
  boot.kernel.sysctl."net.ipv4.ip_forward" = true;
  virtualisation.docker = {
    enable = true;

    rootless = {
      enable = true;
      setSocketVariable = true;
    };
  };

  nix.settings.trusted-users = [ "root" ];

  services.gitlab-runner = {
    enable = true;
    clear-docker-cache.enable = true;
    settings.concurrent = 3;

    # runner for building in docker via host's nix-daemon
    # nix store will be readable in runner, might be insecure
    services.native-runner = {
      # community managed, automatically updated nix image with flakes + commands pre-enabled
      dockerImage = "nixpkgs/nix-flakes:nixos-${config.system.nixos.release}-${pkgs.system}";
      dockerVolumes = [
        # the items are ro because we write to the store via the daemon
        "/nix/store:/nix/store:ro"
        "/nix/var/nix/db:/nix/var/nix/db:ro"
        "/nix/var/nix/profiles/system/etc/ssl/:/etc/ssl/:ro"
        "/nix/var/nix/daemon-socket:/nix/var/nix/daemon-socket:ro"
        "${pkgs.bash}/bin/bash:/usr/bin/sh:ro"
        "${pkgs.bash}/bin/bash:/bin/bash:ro"
        "${pkgs.bash}/bin/bash:/usr/bin/bash:ro"
        "${pkgs.bash}/bin/bash:/bin/sh:ro"
      ];
      dockerDisableCache = true;
      registrationFlags = [
        "--docker-pull-policy=if-not-present"
        "--docker-allowed-pull-policies=if-not-present"
        "--docker-allowed-pull-policies=always"
      ];
      environmentVariables = {
        # we use the shared nix daemon of the host
        NIX_REMOTE = "daemon";
        ENV = "/etc/profile";
        USER = "root";
        # NOTE: we override the nix installation in the container because
        # it is linked to the original nix store from the container.
        # However this store, nor the dynamic libraries in it, can be found
        # because of the overlay mount. Also ensure the nix version is
        # synced to the system nix daemon.
        PATH =
          (pkgs.lib.strings.makeSearchPathOutput "bin" "bin" (
            with pkgs;
            [
              gnugrep
              coreutils
              nix
              openssh
              gitleaks
              bash
              git
            ]
          ))
          + ":/nix/var/nix/profiles/default/bin:/usr/local/bin:/usr/local/sbin:/nix/var/nix/profiles/default/sbin:/bin:/sbin:/usr/bin:/usr/sbin";
      };

      authenticationTokenConfigFile = <path to token file, use sops-nix or agenix here>;
    };
  };
}

Plain runner

You can also use the daemon inside the container, the advantages are:

proper ressource limits, you can limit CPU and memory per job
runners can be trivially reused for other images (can be reliably used for untagged jobs)

The disadvantages are:

caching requires extra work, see "Reducing duplicate work" for strategies to address this
sandboxing is only possible for priviledged runners

A sample configuration for a plain runner is:

{ pkgs, config, ... }: {
  boot.kernel.sysctl."net.ipv4.ip_forward" = true; # 1
  virtualisation.docker = {
    enable = true;
    enableOnBoot = true;
    autoPrune.enable = true;
  };

  services.gitlab-runner = {
    enable = true;
    settings = {
      concurrent = 4;
      listen_address = "127.0.0.1:9252";
    };

    services.runner = {
      # community managed, automatically updated nix image with flakes + commands pre-enabled
      dockerImage = "nixpkgs/nix-flakes:nixos-${config.system.nixos.release}-${pkgs.system}";

      dockerVolumes = [
        # passthrough bash & grep for gitlab ci (used inside the executor, not contained in the base image)
        "${lib.getExe pkgs.pkgsStatic.gnugrep}:/usr/bin/grep:ro"
        "${lib.getExe pkgs.pkgsStatic.bash}:/usr/bin/sh:ro"
        "${lib.getExe pkgs.pkgsStatic.bash}:/usr/bin/bash:ro"
      ];

      registrationFlags = [
        "--docker-pull-policy=if-not-present"
        "--docker-allowed-pull-policies=if-not-present"
        "--docker-allowed-pull-policies=always"
      ];

      authenticationTokenConfigFile = ;
    };
    };
  };
}

If you want to use sandboxing and are fine with a priviledged runner, add the extra flags and volumes below:

{ config, pkgs, ... }:
let
  nixJoin = list: builtins.concatStringsSep " " list;
  # enable sandbox & flakes + passthrough of host substituters
  nix-conf = pkgs.writeText "nix.conf" ''
    accept-flake-config = true
    experimental-features = nix-command flakes
    max-jobs = auto
    sandbox = true

    substituters = ${nixJoin config.nix.settings.substituters}
    trusted-public-keys = ${nixJoin config.nix.settings.trusted-public-keys}
    extra-substituters = ${nixJoin config.nix.settings.extra-substituters}
    extra-trusted-public-keys = ${nixJoin config.nix.settings.extra-trusted-public-keys}
  '';
in
{
  services.gitlab-runner.services.runner = {
    dockerVolumes = [
      # inject host nix conf for substituters etc. into container
      # this ensures our nix cache get's used for all devshells
      "${nix-conf}:/etc/nix/nix.conf:ro"
      # passthrough bash & grep for gitlab ci (used inside the executor, not contained in the base image)
      ...
    ];
    registrationFlags = [ "--docker-privileged" ];
  };
}

Please also note that adjusting the nix.conf with dockerVolumes also can enable you to adjust the default trusted-substituter inside the container. This can be useful to inject your own public cache even if you don't enable sandboxing.

Runners with S3 cache

The official documentation is a bit sparse here, so this serves as an extension to both runners types. When you have runners on multiple hosts you likely want a shared cache between runners to exchange data between jobs reliably. For the GitLab runner NixOS module this means you will have to use the CLI registration flags instead of the YAML config, below are the relevant flags:

{ ... }:
{
  services.gitlab-runner.services.runner.registrationFlags = [
    "--cache-s3-server-address=s3.example.com"
    "--cache-s3-access-key=$(cat ${access_key.path})"
    "--cache-s3-secret-key=$(cat ${secret_key.path})"
    "--cache-s3-bucket-name=$(cat ${bucket_name.path})"
    "--cache-s3-bucket-location=us-east-1"
    "--cache-s3-authentication_type=access-key"
    "--cache-type=s3"
    "--cache-shared"
  ];
}

:::warning These flags are written to the store, using --cache-s3-secret-key=$(cat ${path to secret}) with agenix or sops-nix is strongly recommended. :::

Both garage and minio are suitable for self-hosting this s3 cache and available as NixOS modules.

Tips: Reducing duplicate work

GitLab CI can cache some artifacts but will not cache paths outside the project directory. This notably affects the nix store.

If you have many jobs or a complex devshell you will likely spend a significant amount of time just downloading, or worse building, artifacts before even starting the actual job task, e.g., running unit tests. Below are two ways that have worked for me:

Specialized container images

Existing images

If you have 10 jobs that need, e.g., devshell, it is very likely an improvement to use an image where it is already available instead of downloading it on every build. The advantage here is that on each run you won't have to download/build the artifact but can instead start directly (ot at least faster).

Some useful image collections are:

github:nix-community/docker-nixpkgs: Images for nix and common nix development tooling (cachix, devenv, devcontainer, ...)
dockerhub nixos/nix: Official nix container image from github:NixOS/nix

Building images

You can also build your own container images with nix, see the following documentation for guidance on this topic:

nix.dev -- guide on building docker images
dockerTools reference documentation
nix2container -- archive-less builder (can be faster than dockerTools)

You can upload docker images built with nix to the GitLab container registry with, e.g., skopeo. This can either be done on a per project/repository basis or, e.g., in a shared image repository with regularly built images based on scheduled pipelines. The latter may be useful when dealing with multiple repositories that share the same tooling.

Streaming image into GitLab registry

:::info This is an adjusted example from the reference documentation. It assumes you want to upload an image to the GitLab registry with name builder and tag latest. :::

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs =
    { self, nixpkgs }:
    let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in
    {
      packages.x86_64-linux.image = pkgs.dockerTools.streamLayeredImage {
        name = "hello";
        contents = [ pkgs.hello ];
      };
    };
}

With the job:

build-image:
  script:
    - echo "$CI_JOB_TOKEN" | skopeo login --insecure-policy "$CI_REGISTRY" -u gitlab-ci-token --password-stdin
    - $(nix build --no-link --print-out-paths .#image) | gzip --fast | skopeo copy docker-archive:/dev/stdin "docker://$CI_REGISTRY_IMAGE/builder:latest"

This job will in echo ... log into the registry with the ephemeral job token, this is required for the upload from the runner. In $(nix ...) the image is built and streamed to stdout, afterward it is compressed (gzip --fast) and pushed to the registry (skopeo ...).

Using binary caches

Nix binary caches allow you to cache the built results of an evaluated derivation.

In a CI/CD pipeline these may be used to push the results of a build to enable derivation-level caching of results. This means you would, e.g., build an artifact in a job, push it to the cache and let successive jobs download it from the cache again (instead of building it).

In my experience, with the constraint of self-hosting most things, attic is a good fit for this. But, especially if you want commercial support/a hosted non-free offering, cachix is a popular solution.

It should be noted that neither have particularly good GitLab support and appear to be mainly used with GitHub. They are however both compatible but you may have to put in some extra work.

attic

attic authenticates with to the server with a signed token. If you plan to use this token in a pipeline, give it push/pull permissions and store it in a CI variable and adjust permissions as needed.

In a job you first need to log in and tell it to configure your local nix.conf:

build-stuff:
  before_script:
    - attic login upstream https://cache.example.com "$ATTIC_TOKEN"
    - attic use upstream:<your cache>
  script:
    - nix build .#cake

You can then either push a derivation output (extending the example from above):

build-stuff:
  after_script:
    - attic push upstream:<your cache> ./result

This will make the next run[^inputs] fetch .#cake from the cache instead of (re)building the derivation. It may also be helpful to use rules if you only want to rebuild on particular changes.

[^inputs]: Assuming the same inputs were used and the cache has not been garbage collected.

:::note Pushing the result of a build will likely not include all dependencies of the build. :::

This may be extended by pushing the direct dependencies:

build-stuff:
  - attic push ./result upstream:<your cache>
  - nix-store --query --references ./result | attic push upstream:<your cache> --stdin

Or the whole store,

build-stuff:
  - nix path-info --all | attic push upstream:<your cache> --stdin

:::warning Unless you know you need this, don't push the whole store. It will also contain derivations already included in the image and may bloat your cache. :::

Tips: Scripting

Your pipelines will often contain some sort of scripts for, e.g., the image upload from above. These scripts can be made reusable as nix packages.

One of the most helpful parts for small scripts are the Trivial build helpers, in particular writeShellApplication. They enable the concise packaging of small scripts.

The main advantage of this approach is the easy invocation of commands locally. It also allows you to, e.g. with writeShellApplication, run shellcheck over your CI scripts.

Extended example

Building on the example from Streaming image into GitLab registry:

{
  inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

  outputs =
    { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
    in
    {
      packages.${system} = {
        image = pkgs.dockerTools.streamLayeredImage {
          name = "hello";
          contents = [ pkgs.hello ];
        };
        upload-image = pkgs.writeShellApplication {
          name = "upload-image";
          runtimeInputs = [ pkgs.skopeo pkgs.gzip ];
          text = ''
            echo "$CI_JOB_TOKEN" | skopeo login --insecure-policy "$CI_REGISTRY" -u gitlab-ci-token --password-stdin
            ${self.packages.${system}.image} | gzip --fast | skopeo copy docker-archive:/dev/stdin "docker://$CI_REGISTRY_IMAGE/builder:latest"
          '';
        };
      };
    };
}

Which can now be used as:

build-image:
  script:
    - nix run .#upload-image

And you can of course also then invoke nix run .#upload-image locally and test with, e.g., a PAT instead of a job token when an issue occurs.

Tips: Flake Evaluation caching

:::warning Below is experimental, it led to a notable speedup in my pipelines but may have side effects. :::

GitLab CI cannot currently cache paths outside the project directory. This does not just affect the nix store but also the flake eval cache.

A workaround is to store the cache from $NIX_CACHE_HOME in the after_script inside the project directory and preseed $NIX_CACHE_HOME in the before_script in the same manner.

cached-eval:
  before_script:
    - if [ ! -d .cache ]; then mkdir .cache; fi
    - export NIX_CACHE_HOME="$HOME/cache"
    - mv .cache "$NIX_CACHE_HOME"
  script:
    - nix build .#...
  after_script:
    - mv "$HOME/cache" .cache

Tips: KVM

:::note Passing through the KVM device obviously requires a working KVM device on the host. This requires a CPU with Intel VT-x or AMD SVM on a bare-metal host or nested virtualisation for VMs.

To check if the device is present, check if /dev/kvm exists. If it doesn't, but your hardware should support it, ensure that the corresponding kernel module kvm_intel or kvm_amd is loaded. :::

:::warning As far as I know, this only works with priviledged runners. :::

Running NixOS tests efficiently inside a builder requires the builder feature kvm. As the name implies, this means /dev/kvm must be present inside the container AND the kernel module must be loaded.

To pass through the device, use:

services.gitlab-runner.services.${runner}.registrationFlags = [
  "--docker-devices=/dev/kvm"
];

Under most circumstances, the kernel module is loaded by book.kernelPackages when generated from nixos-generate-config. If it isn't, add kvm-amd or kvm-intel to boot.kenelPackages in your host config.