Cloudflare R2 as Nix binary cache

Practical for Flake (workstations + box + GitHub CI): free egress, ~$0 on the 10 GB free tier until closures grow, no always-on cache server required.

Fit for Noa’s flake

Flake (pelikanade/flake) is a personal NixOS flake with multiple hosts, Home Manager, and agent tooling. A private R2 cache fits well:

Push from CI (and optionally from just build / a post-build-hook on box) so Parallax / Asymmetry / box share closures without rebuilding.
Public HTTPS pull via a custom domain keeps client config to substituters + trusted-public-keys — no AWS creds on laptops.
sops-nix already holds secrets: store the R2 access key + Nix signing secret there (or GH Actions secrets for CI only).
Skip caching Tern (requireFile / never-cached) and anything you treat as non-substitutable.
If free-tier storage fills up: either filter uploads (don’t mirror cache.nixos.org paths) or graduate to niks3 / Attic on R2 for reference-aware GC.

1. Minimal working config

Bucket + API token

Create an R2 bucket (Standard storage; jurisdiction/location as you prefer).
Create an R2 API token scoped to that bucket with Object Read & Write (Cloudflare dashboard → R2 → Manage R2 API Tokens). Yinfeng’s Terraform uses the permission group Workers R2 Storage Bucket Item Write (homemade S3 cache).
Note the S3 API endpoint: https://<ACCOUNT_ID>.r2.cloudflarestorage.com (R2 S3 API). Region for the S3 API is auto.

Signing key

nix key generate-secret --key-name cache.example.org-1 > ./cache.secret
nix key convert-secret-to-public < ./cache.secret   # → cache.example.org-1:BASE64…

(nix key generate-secret)

Store URL (push)

From the Nix S3 Binary Cache Store manual:

s3://nix-cache?endpoint=<ACCOUNT_ID>.r2.cloudflarestorage.com&region=auto&scheme=https&compression=zstd

Useful query params (same manual; defaults in parentheses):

Param	Role	Default / notes
`endpoint`	R2 account endpoint (no `https://` in some docs; Nix accepts host or URL depending on version — prefer host form as in MinIO examples)	empty → AWS
`region`	Must be `auto` for R2	`us-east-1`
`scheme`	`https`	`https`
`addressing-style`	`auto` uses path-style for custom endpoints	`auto`
`profile`	`~/.aws/credentials` profile	`default`
`compression`	`xz` / `bzip2` / `gzip` / `zstd` / `none`	`xz`
`parallel-compression`	multi-thread xz/zstd	`false`
`ls-compression`	compression for `.ls` listings	empty
`write-nar-listing`	write JSON NAR listing	`false`
`multipart-upload`	large NAR multipart	`false`
`multipart-threshold`	when multipart kicks in	`100 MiB`
`multipart-chunk-size`	part size (≥ 5 MiB)	`5 MiB`
`secret-key` / `secret-keys`	sign on upload	empty

Credentials

Nix uses the AWS default credential chain:

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_EC2_METADATA_DISABLED=true   # important off-AWS (R2); avoids IMDS stalls

Or a profile:

[r2]
aws_access_key_id = ...
aws_secret_access_key = ...

then ?profile=r2 on the store URL.

Push

nix store sign --recursive --key-file ./cache.secret ./result
nix copy ./result --to \
  's3://nix-cache?endpoint=<ACCOUNT_ID>.r2.cloudflarestorage.com&region=auto&scheme=https&compression=zstd'

Or sign via store URI secret-key= / daemon secret-key-files so uploads are signed automatically.

NixOS / nix.conf (pull — HTTPS)

After connecting a custom domain to the bucket (below):

{
  nix.settings = {
    substituters = [
      "https://cache.example.org?priority=30"
      "https://cache.nixos.org"
    ];
    trusted-public-keys = [
      "cache.example.org-1:BASE64…"
      "cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
    ];
  };
}

2. CI recipe (GitHub Actions)

Pattern from asa1984/binary-cache-example (R2-backed; public at cache.asa1984.dev):

name: Copy package to S3 binary cache store

env:
  AWS_PROFILE_NAME: builder
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  BINARY_CACHE_SECRET_KEY: ${{ secrets.BINARY_CACHE_SECRET_KEY }}
  S3_API_ENDPOINT: ${{ secrets.S3_API_ENDPOINT }}  # <accountid>.r2.cloudflarestorage.com

on:
  push:
    paths-ignore: ["**/README.md"]

jobs:
  copy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: DeterminateSystems/nix-installer-action@main

      - name: Build package
        run: nix build . --accept-flake-config

      - name: Sign package with secret key
        run: |
          echo "$BINARY_CACHE_SECRET_KEY" > ./secret.key
          nix store sign --recursive --key-file ./secret.key ./result

      - name: Configure AWS credentials
        run: |
          nix shell nixpkgs#awscli --command aws configure set aws_access_key_id "$AWS_ACCESS_KEY_ID" --profile "$AWS_PROFILE_NAME"
          nix shell nixpkgs#awscli --command aws configure set aws_secret_access_key "$AWS_SECRET_ACCESS_KEY" --profile "$AWS_PROFILE_NAME"

      - name: Copy package
        run: |
          export AWS_EC2_METADATA_DISABLED=true
          nix copy ./result --to \
            "s3://nix-cache?profile=${AWS_PROFILE_NAME}&endpoint=${S3_API_ENDPOINT}&region=auto&scheme=https&compression=zstd"

`post-build-hook` (builder / Hydra-style)

On a trusted builder, upload every local build:

# nix.conf
secret-key-files = /etc/nix/cache.secret
post-build-hook = /etc/nix/upload-to-cache.sh

#!/usr/bin/env bash
set -eu
set -f
export IFS=' '
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_EC2_METADATA_DISABLED=true
nix copy --to 's3://nix-cache?endpoint=…&region=auto&scheme=https&compression=zstd' $OUT_PATHS

(chmod +x the hook — a common footgun on Discourse.) Hook runs only for locally built paths; remote-builder outputs need signing/upload on the builder or a separate nix copy pass.

3. Public pull via custom domain

Why HTTPS, not `s3://`, for clients

The S3 store manual recommends: if the cache is public, use the HTTP Binary Cache Store (https://…) so clients skip credential lookup. Authenticated s3:// pulls work but force every machine to carry R2 keys.

Enable public access

From Public buckets:

Prefer a custom domain on the same Cloudflare account as the bucket (production).
*.r2.dev is fine for experiments only — rate-limited, no Cache / WAF / Bot Management.
Custom domain ⇒ Cloudflare CDN in front of R2. Default cache only covers certain extensions; .narinfo / many NAR types need a Cache Rule (“Cache Everything” / eligible for cache) (Enable cache in an R2 bucket, Default Cache Behavior).

What Nix expects

GET https://cache.example.org/nix-cache-info
GET https://cache.example.org/<hash>.narinfo
GET https://cache.example.org/nar/<hash>.nar.zst   # URL from narinfo

nix-cache-info typically:

StoreDir: /nix/store
WantMassQuery: 1
Priority: 30

Caching gotchas (404s + narinfo)

Layer	Behavior	Mitigation
Cloudflare edge	Without origin `Cache-Control`, 404/410 default Edge TTL is 3 minutes (Default Cache Behavior)	Short negative TTL is usually OK; avoid caching 404s for long via Cache Rules if you push often
Nix client	`narinfo-cache-negative-ttl` default 3600 s; positive ~30 d (NixOS/nix#781, conf-file)	After a push that clients probed too early: `--refresh` / `--option narinfo-cache-negative-ttl 0`, or wipe `~/.cache/nix/binary-cache-v.sqlite`
NARs	Content-addressed → safe to cache long (`immutable`)	Cache Rule: long TTL for `/nar/*`
narinfo	Can gain new `Sig:` / uploads	Moderate TTL (hours–1d), not years

Authenticate pulls only if the bucket stays private: keep clients on s3://… with read keys, or put Attic/niks3/Worker auth in front (cf-contrib/nix-cache, T4ko0522/cf-edgeNix).

4. Gotchas (Nix / Lix / Determinate / aws-sdk)

R2 S3 dialect

Endpoint: <accountid>.r2.cloudflarestorage.com; region auto (R2 S3 API).
Nix addressing-style=auto → path-style for custom endpoints (correct for R2).
Set AWS_EC2_METADATA_DISABLED=true off AWS.

Default checksum headers (Jan 2025 wave)

AWS SDKs started sending default integrity checksums (x-amz-checksum-* / x-amz-sdk-checksum-algorithm). R2 initially rejected them (Header 'x-amz-checksum-algorithm' … not implemented). Cloudflare tracked this as status incident t5nrjmpxc1cj / community thread (Jan–Feb 2025). Workarounds elsewhere: request_checksum_calculation=when_required (R2 AWS examples). R2 now documents partial checksum matrix (CRC32/CRC32C/SHA composite, etc.) on the S3 API page.

Nix-specific: Nix’s S3 uploader also attaches Content-MD5 (see current s3-binary-cache-store.cc). Combinations of MD5 + SDK default CRC headers caused InvalidRequest: You can only specify one non-default checksum at a time in other tools (e.g. Vector). If an older Nix built against a “checksums-by-default” aws-sdk-cpp hits R2 upload failures, upgrade Nix / aws-sdk-cpp or prefer a post–curl-S3 Nix.

aws-sdk-cpp multipart / download bugs

Issue	Symptom	Versions / fix
NixOS/nix#12671	`InvalidChunkSizeError` on multipart upload	aws-sdk-cpp < 1.11.445; seen with `multipart-upload=true`
NixOS/nix#13984 / nixpkgs#442740	`Response checksums mismatch` fetching multipart objects (`x-amz-checksum-type: COMPOSITE`)	aws-sdk-cpp 1.11.612 broken; ≥ 1.11.615 fixed; Nix 2.28–2.30 on affected nixpkgs pins
NixOS/nix#13752	curl-based S3 (drops aws-sdk-cpp for S3)	Merged 2025-10-15 — Nix master / releases after this avoid the SDK class of bugs for S3 I/O

Practical version notes (2026-10):

Nix ≥ ~2.33+ (post curl-S3): preferred for R2 push/pull via s3://.
Older Nix / nixpkgs-packaged Nix still on aws-sdk-cpp: pin sdk ≥ 1.11.615 or avoid multipart until upgraded; watch checksum errors on R2.
Lix: no dedicated R2/checksum issues turned up in lix-project search; treat similarly if it still links aws-sdk-cpp for S3.
Determinate Nix (e.g. 3.22.x / upstream Nix 2.35.x per X post): should include modern S3 stack; use their S3 store docs.

Other operational gotchas

Don’t lifecycle-delete NARs blindly while narinfos still point at them (broken substitutes). Prefer Attic/niks3 GC or delete narinfo first, then unreferenced NARs.
nix copy uploads the full closure you ask for — it will happily store paths already on cache.nixos.org and burn R2 quota. Filter uploads or use an overlay (Yinfeng’s nix-cache-overlay).
Signing is mandatory for untrusted substituters: unsigned paths are ignored unless trusted-substituters + require-sigs = false (don’t do that lightly).

5. Cost and alternatives

R2 pricing (official, as of docs fetched 2026-10)

Meter	Free tier / month	Then
Storage (Standard)	10 GB	$0.015 / GB-month
Class A (mutate/list)	1 M	$4.50 / M
Class B (read)	10 M	$0.36 / M
Egress	—	Free

Infrequent Access is cheaper storage but adds retrieval + higher op prices — usually a poor fit for a hot binary cache. Class A volume from nix copy narinfo probing can dominate before storage does.

Brief comparison

Option	Model	Pros	Cons
Raw R2 + `nix copy`	DIY S3 protocol	Zero egress, serverless, CDN pull, full control	No GC intelligence; Class A chatter; you own signing/CI
Cachix	Hosted	Dead-simple push/pull, GC, team features	Paid beyond free tier; egress/ops elsewhere
Attic (zhaofengli/attic)	Self-host; S3/R2 backend	Multi-tenant, chunk dedup, GC, server-side signing	Runs atticd + DB; some reports of slow uploads vs raw S3
niks3	GC server + direct S3 reads	Recommends R2; clients read bucket/CDN; ref-tracking GC; OIDC for GHA	Still a small server + Postgres for GC metadata
harmonia	Serve local `/nix/store` over HTTP	Fast Rust nix-serve replacement	Needs a machine with the store; not object storage
nix-serve / nix-serve-ng	Classic HTTP cache	Simple	Single host; no R2
FlakeHub Cache	Hosted (Determinate)	Integrates with flakes / Determinate Nix	Subscription ($7+/mo personal tier historically)
Garnix	CI + cache	Builds for you	Trust/CI product, not a dumb bucket
Workers + R2 (cf-contrib/nix-cache, cf-edgeNix)	Edge HTTP API	Auth, validation, edge cache, optional server signing	More moving parts than raw `s3://`

Recommendation for Noa: start with raw R2 + custom domain; move to niks3 or Attic on R2 when GC/retention or upload filtering becomes painful.

6. GC / retention

R2 lifecycle rules

Object lifecycles can expire by prefix/age or abort incomplete multipart uploads (default abort incomplete MPU after 7 days).

Danger: time-based delete of nar/* while *.narinfo still references those files → clients get narinfo hits then NAR 404s. Prefer:

Delete or rewrite narinfo first, then remove unreferenced NARs; or
Use Attic GC (LRU / retention period in config-template.toml) or niks3 GC (Postgres reference tracking + configurable retention / pins).

What “GC” means here

Local nix-collect-garbage does not clean R2. Binary-cache GC is a separate problem: either never delete (fine while under 10 GB), lifecycle with care, or a tool that understands narinfo → NAR references.

7. Compression & large NARs

From the S3 store settings:

# Prefer zstd for personal caches (speed/size); xz is Nix's default
nix copy ./result --to 's3://nix-cache?…&compression=zstd&parallel-compression=true'

# Optional listings
nix copy ./result --to 's3://nix-cache?…&write-nar-listing=true&ls-compression=zstd'

# Large closures
nix copy ./result --to 's3://nix-cache?…&multipart-upload=true&multipart-threshold=104857600'

parallel-compression is only for xz and zstd. Multipart is off by default; enable for big NARs (and use a Nix new enough that multipart is reliable on R2).

8. Field notes

Configs and write-ups people actually use:

Yinfeng (2026): public R2 + custom domain cache.li7g.com, Terraform for bucket/domain/token, nix copy + optional nix-cache-overlay to skip paths present upstream — homemade Nix S3 cache.
lgug2z (2024): Attic on Fly.io with R2 backend (region = "auto", endpoint = "https://<id>.r2.cloudflarestorage.com") — article / HN 86 pts.
asa1984: minimal GHA → R2 → custom domain — binary-cache-example.
Attic template explicitly documents R2-style endpoints — config-template.toml.
niks3 docs recommend Cloudflare R2; Numtide/Clan run it in production — Mic92/niks3, Discourse 1.5 release.
NixOS Wiki Attic + Caddy Cache-Control patterns for narinfo/NAR — Binary Cache.
X: fmzakari asking about R2+Nix (2024); asa_high_ost wanting MinIO tutorial on R2; HN bots circulating the Attic+R2 post; T4ko0522 on owning a CF-deployed nix cache (2026).

9. Sources

Manuals & Cloudflare

GitHub issues / PRs

NixOS/nix#13752 — curl-based S3 (merged 2025-10-15)
NixOS/nix#13984 / nixpkgs#442740 — COMPOSITE checksum mismatch
NixOS/nix#12671 — InvalidChunkSizeError multipart
NixOS/nix#781 — negative narinfo caching
zhaofengli/attic / docs.attic.rs
Mic92/niks3
asa1984/binary-cache-example
cf-contrib/nix-cache

Blogs / Discourse / HN / X

Gaps

No strong Reddit hits for “R2 + Nix cache” via Exa site:reddit.com in this pass (HN + Discourse + blogs denser).
Lix tracker returned no R2/checksum-specific hits; Lix behavior inferred from shared aws-sdk / S3 heritage, not a Lix ticket.
Exact minimum Nix release number that first ships curl-based S3 in a stable tag was not pinned beyond “merged to master 2025-10-15 / present in current 2.33+ manuals and tree”; verify with nix --version + release notes before relying on it in CI.
Cachix / Garnix dollar amounts change often — treat comparison as architectural, re-check vendor pages before budgeting.

Cloudflare R2 as Nix binary cache

Cloudflare R2 as Nix binary cache

Fit for Noa’s flake

1. Minimal working config

Bucket + API token

Signing key

Store URL (push)

Credentials

Push

NixOS / nix.conf (pull — HTTPS)

2. CI recipe (GitHub Actions)

post-build-hook (builder / Hydra-style)

3. Public pull via custom domain

Why HTTPS, not s3://, for clients

Enable public access

What Nix expects

Caching gotchas (404s + narinfo)

4. Gotchas (Nix / Lix / Determinate / aws-sdk)

R2 S3 dialect

Default checksum headers (Jan 2025 wave)

aws-sdk-cpp multipart / download bugs

Other operational gotchas

5. Cost and alternatives

R2 pricing (official, as of docs fetched 2026-10)

Brief comparison

6. GC / retention

R2 lifecycle rules

What “GC” means here

7. Compression & large NARs

8. Field notes

9. Sources

Manuals & Cloudflare

GitHub issues / PRs

Blogs / Discourse / HN / X

Gaps

`post-build-hook` (builder / Hydra-style)

Why HTTPS, not `s3://`, for clients