Nix / NixOS @programming.dev secana @programming.dev 3mo ago

Issue with transparent cache.nixos.org proxy

Hi! I would like to host a transparent proxy for cache.nixos.org on my local kubernetes cluster.

I took the following NGINX config https://nixos.wiki/wiki/FAQ/Private_Cache_Proxy and created all the folders on the mounted storage.

This is the kubernetes deployment:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  nginx.conf: |
    server {
      listen 80;
      server_name nix-cache.raspi.home;

      location ~ ^/nix-cache-info {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nix-cache-info/temp;
        root               /data/nginx/nix-cache-info/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }

      location ~^/nar/.+$ {
        proxy_store        on;
        proxy_store_access user:rw group:rw all:r;
        proxy_temp_path    /data/nginx/nar/temp;
        root               /data/nginx/nar/store;

        proxy_set_header Host "cache.nixos.org";
        proxy_pass https://cache.nixos.org;
      }
    }

To use the cache I added it to the substituters.

  nix.settings.substituters = [
    "https://nix-cache.raspi.home/"
  ];

But when I try to use it, get the error:

# Trigger a download
nix develop nixpkgs#just
# Error message
warning: 'https://nix-cache.raspi.home' does not appear to be a binary cache

In the logs of the NGINX I see the following error:

2024/08/03 12:09:30 [error] 31#31: *3 open() "/usr/share/nginx/html/nix-cache-info" failed (2: No such file or directory), client: 10.42.2.7, server: localhost, request: "GET /nix-cache-info HTTP/1 │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "GET /nix-cache-info HTTP/1.1" 404 153 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"                                                                             │
│ 10.42.2.7 - - [03/Aug/2024:12:09:30 +0000] "PUT /nix-cache-info HTTP/1.1" 405 157 "-" "curl/8.8.0 Nix/2.18.5" "10.42.2.1"

Any ideas whats wrong? I'm neither an nix nor an nginx expert, so maybe it is something really simple but I cannot figure it out.

4 comments

/usr/share/nginx/html/nix-cache-info

That indicates your nginx is probably still using the default config.

This is the default config in nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

Reading the dockerhub page for your image it states

Customize configuration

You can mount your configuration file, or build a new image with it.

If you wish to adapt the default configuration, use something like the following to get it from a running nginx container:

$ docker run --rm --entrypoint=cat nginx /etc/nginx/nginx.conf > /host/path/nginx.conf

And then edit /host/path/nginx.conf in your host file system.

Basically

volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/sites-available/default # this should be /etc/nginx/nginx.conf

You might want to take some configuration options from the default config like the user, worker connections, etc.

Good luck.

Anti Commercial-AI license

Thanks for the response. You are right, the config was at the wrong path. Unfortunately, the config itself does not work, too. After a bit of testing around this worked for me:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: nix-cache-volume
spec:
  capacity:
    storage: 500Gi
  storageClassName: manual
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: "/mnt/k8s/nix-cache" # Needs exists before PV is created!
  persistentVolumeReclaimPolicy: Retain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nix-cache-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: manual
  resources:
    requests:
      storage: 500Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nix-cache
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nix-cache
  template:
    metadata:
      labels:
        app: nix-cache
        name: nix-cache
    spec:
      volumes:
        - name: nix-cache-storage
          persistentVolumeClaim:
            claimName: nix-cache-pvc
        - name: nix-cache-config
          configMap:
            name: nix-cache-config
      containers:
        - name: nix-cache
          image: nginx:1.27.0 
          ports:
            - containerPort: 80
          volumeMounts:
            - name: nix-cache-storage
              mountPath: /data
            - name: nix-cache-config
              mountPath: /etc/nginx/nginx.conf
              subPath: nginx.conf
              readOnly: true
          resources:
            limits:
              memory: "512Mi"
              cpu: "300m"
            requests:
              memory: "256Mi"
              cpu: "200m"
---
apiVersion: v1
kind: Service
metadata:
  name: nix-cache
spec:
  selector:
    app: nix-cache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nix-cache-ingress
  annotations:
    traefik.ingress.kubernetes.io/router.tls: "true"
spec:
  rules:
    - host: "nix-cache.raspi.home"
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: nix-cache
                port:
                  number: 80
  tls:
    - secretName: nix-cache-raspi-home-tls
      hosts:
        - "nix-cache.raspi.home"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: nix-cache.raspi.home
spec:
  commonName: nix-cache.raspi.home
  dnsNames:
    - "nix-cache.raspi.home"
  secretName: nix-cache-raspi-home-tls
  issuerRef:
    name: ca-issuer
    kind: ClusterIssuer
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nix-cache-config
data:
  # See: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit
  nginx.conf: |
    events {
        worker_connections 1024;
    }
    http {
      proxy_cache_path /data/nginx/cache max_size=500G keys_zone=cache_zone:50m inactive=365d;
      proxy_cache cache_zone;
      proxy_cache_valid 200 365d;
      proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_504 http_403 http_404 http_429;
      proxy_ignore_headers X-Accel-Expires Expires Cache-Control Set-Cookie;
      proxy_cache_lock on;

      server {
          listen 80;

          server_name nix-cache.raspi.home;

          location /nix-cache-info {
              return 200 "StoreDir: /nix/store\nWantMassQuery: 1\nPriority: 41\n";
          }

          location / {
              proxy_set_header Host $proxy_host;
              proxy_pass https://cache.nixos.org;
          }
      }
    }

The config is an adaption from this blog post: https://www.channable.com/tech/setting-up-a-private-nix-cache-for-fun-and-profit

Nice. Glad you got it working 👍

Anti Commercial-AI license

Might be misunderstanding this myself, but I thought a transparent proxy intercepts the requests to the destination.

Now if this is the case, I wouldn't expect any changes to your nix config, as it should be transparently intercepting the requests to cache.nixos.org. Now how this would work with nginx in this context I haven't quite gotten my head around.

Unless your set your gateway to be your nginx box; to handle all requests and forward them on as necessary.

Might give this a go on my build server if I get a chance.