total{debug} total{debug}
>_ GCP · Automation

A serverless signed APT repository on GCP

Steven Marks
Steven Marks
24 Aug 2026 · 23 min read
A serverless signed APT repository on GCP

You’ve built a .deb. Now you need a way for people to easily install it: apt install example-agent, upgrades included, no curl | sudo bash scripts so it’s easier for you to manage.

The usual answer is to stand up a repository server. A VM running reprepro or aptly behind nginx, with a certificate, a firewall, patches, and a pager rotation. That is a lot of standing infrastructure for something that is, when you look closely, a folder of files.

Because that is all an APT repository is, a specific directory layout, a couple of generated index files and one signature over the top. It is static content, so it does not need a server at all. It needs somewhere to store files, something to serve them over TLS, and something to regenerate the indexes when a package changes.

On GCP that maps cleanly onto a storage bucket, a load balancer, and a Cloud Run job. Nothing runs between publishes. This post builds the whole thing, and then fixes the one part that most people get subtly wrong: the reindex.

What an APT repository actually is

Two directory trees and a signature.

The pool holds the packages. The exact .deb files, laid out by name so they never collide:

1
2

pool/stable/main/e/example-agent/example-agent_1.0.0_amd64.deb
pool/stable/main/e/example-agent/example-agent_1.1.0_amd64.deb

The dists tree holds the metadata that apt reads first:

1
2
3
4
5

dists/stable/main/binary-amd64/Packages       # one stanza per package
dists/stable/main/binary-amd64/Packages.gz     # the same, compressed
dists/stable/Release                            # checksums of the Packages files
dists/stable/InRelease                          # Release, clear-signed
dists/stable/Release.gpg                        # detached signature of Release

A single Packages stanza is just the package’s control fields plus its location and hashes:

1
2
3
4
5
6
7

Package: example-agent
Version: 1.1.0
Architecture: amd64
Filename: pool/stable/main/e/example-agent/example-agent_1.1.0_amd64.deb
Size: 4194
MD5sum: ...
SHA256: ...

The trust chain runs top down. You sign Release. Release contains the SHA256 of each Packages file. Each Packages stanza contains the SHA256 of a .deb. So one signature, checked against one public key on the client, vouches for every byte the client downloads. That is the whole security model and it is why the signing step is not optional.

The shape of it on GCP

Four pieces, and only one of them ever runs code:

  • A GCS bucket stores pool/ and dists/. This is the repository.
  • A global HTTPS load balancer serves the bucket over TLS on your own domain.
  • A Cloud Run job regenerates and signs the metadata. It runs, then exits.
  • Your release pipeline runs that job after it uploads a package.
flowchart LR
  subgraph PUB["Publishing Pipeline"]
    direction TB
    PIPE["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/pipeline.svg'></td><td><span class='di-t'>Release pipeline</span><br><span class='di-s'>builds .deb artifacts</span></td></tr></table>"]
    JOB["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/cloudrun.svg'></td><td><span class='di-t'>Cloud Run job</span><br><span class='di-s'>indexer</span></td></tr></table>"]
    PIPE -->|"2 · execute"| JOB
  end
  subgraph STORE["Repository Storage"]
    BUCKET["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/bucket.svg'></td><td><span class='di-t'>GCS bucket</span><br><span class='di-s'>pool/ · dists/</span></td></tr></table>"]
  end
  subgraph CLI["Client Access Path"]
    direction LR
    APT["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/apt.svg'></td><td><span class='di-t'>apt client</span><br><span class='di-s'>apt.example.com</span></td></tr></table>"]
    DNS["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/dns.svg'></td><td><span class='di-t'>Cloudflare DNS</span><br><span class='di-s'>CNAME to LB</span></td></tr></table>"]
    LB["<table class='di'><tr><td class='di-ic'><img src='/assets/img/posts/serverless-signed-apt-repository-on-gcp/ic/lb.svg'></td><td><span class='di-t'>HTTPS load balancer</span><br><span class='di-s'>backend bucket + CDN</span></td></tr></table>"]
    APT --> DNS --> LB
  end
  PIPE -->|"1 · upload .deb"| BUCKET
  JOB -->|"3 · index + sign"| BUCKET
  LB -->|"serve dists/ + pool/"| BUCKET

  style PUB fill:#00b2e812,stroke:#00b2e8
  style STORE fill:#f5660012,stroke:#f56600
  style CLI fill:#28c84012,stroke:#28c840

Reads are served by the bucket and the load balancer, which cost nothing to keep idle. Compute exists only for the few seconds it takes to reindex. That is what “serverless” earns you here: there is genuinely no box to keep alive.

The bucket

1
2
3

gcloud storage buckets create gs://example-apt-repo \
  --location=europe-west2 \
  --uniform-bucket-level-access

apt clients are anonymous, so object reads need to be public:

1
2

gcloud storage buckets add-iam-policy-binding gs://example-apt-repo \
  --member=allUsers --role=roles/storage.objectViewer

Everything in the bucket is a public package or a public signature. The private half of the story is the signing key, which never goes near the bucket.

Serving it behind a load balancer

You can point apt straight at https://storage.googleapis.com/example-apt-repo, and it will sort of work, but it will bite you later on with redirects and caching. The clean path is a backend bucket behind a global HTTPS load balancer, on a domain you control.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Managed certificate for the domain
gcloud compute ssl-certificates create apt-cert \
  --domains=apt.example.com --global

# Bucket as a load-balancer backend
gcloud compute backend-buckets create apt-backend \
  --gcs-bucket-name=example-apt-repo

# Route everything to that backend, terminate TLS, get a public IP
gcloud compute url-maps create apt-urlmap --default-backend-bucket=apt-backend
gcloud compute target-https-proxies create apt-proxy \
  --url-map=apt-urlmap --ssl-certificates=apt-cert
gcloud compute forwarding-rules create apt-fr \
  --global --target-https-proxy=apt-proxy --ports=443

Then a DNS A record from apt.example.com to the forwarding rule’s IP.

One gotcha if you sit DNS behind a proxy such as Cloudflare: keep this record unproxied (grey cloud, DNS only). A proxy in front of the repo rewrites requests and caches responses in ways that break apt’s hash checks. apt is fussy about getting exactly the bytes the signature covers, and a helpful CDN in the middle is not helping.

That fussiness has a second consequence worth wiring in now. The .deb bodies are immutable and can cache for a year, but the metadata must never be served stale. If a client reads an old Packages against a fresh Release, it fails with “Hash Sum mismatch” and stops. So the indexer writes dists/ with no-cache, which we will see below.

The signing key

Generate a key dedicated to the repository, export the private half, and put it in Secret Manager. It is the one genuine secret in the system.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

gpg --batch --gen-key <<'EOF'
%no-protection
Key-Type: eddsa
Key-Curve: ed25519
Subkey-Type: ecdh
Subkey-Curve: cv25519
Name-Real: Example Repo Signing
Name-Email: apt@example.com
Expire-Date: 0
%commit
EOF

gpg --armor --export-secret-keys apt@example.com \
  | gcloud secrets create apt-signing-key --data-file=-

Publish the public key next to the repository so clients can install it. The private key only ever exists inside the indexer, in memory, for the length of one run.

The naive indexer, and why it does not scale

Here is the version almost every guide gives you, and the version I shipped first. Sync the pool, scan it, sign it, sync the metadata back:

1
2
3
4
5

gcloud storage rsync -r gs://example-apt-repo/pool pool
dpkg-scanpackages -m -a amd64 pool/stable > dists/stable/main/binary-amd64/Packages
gzip -9c dists/stable/main/binary-amd64/Packages > dists/stable/main/binary-amd64/Packages.gz
apt-ftparchive release dists/stable > dists/stable/Release
# sign, then sync dists back

It works. It is also quietly O(n).

dpkg-scanpackages is stateless. To emit the stanza for a package it opens the .deb, reads its control file, and hashes the whole thing. It has no memory of the last run, so it does this for every package, every time, even the ones that have not changed since the day you added them.

Which means each run has to pull the entire pool down first. A repository with a few hundred packages is already hundreds of megabytes, and it only climbs with every release. To add one 4 MB package you download all of it and rehash every byte. That is a lot of work to learn a single new line of text, and it gets worse as the repo grows: the bigger the pool, the more you re-download to change nothing.

The transfer itself is cheap, in-region and free, so this is not really about cost. It is that the work is proportional to the size of the repository when it should be proportional to the size of the change.

The insight: a stanza is a pure function of a package

Look again at what goes into a Packages stanza: control fields, filename, size, hashes. Every one of those is derived from a single .deb. A .deb in the pool is immutable. Its version and architecture are baked into its filename, and you never rewrite it in place. A new build is a new file.

So a stanza, once computed, is correct forever. There is no reason to ever compute it twice.

That turns the reindex into a cache problem. The Packages file you generated last time is the cache: it already holds a correct stanza for every package the pool contained then. On the next run you only need to compute stanzas for packages that are new since that file was written, and drop stanzas for any that were removed.

The authoritative list of what the pool contains today is one cheap, bodiless call:

1

gcloud storage ls -r "gs://example-apt-repo/pool/stable/**/*.deb"

Listing object names does not download them. So the plan is: list the pool, compare against last time’s Packages, fetch only the genuinely new .deb files, and splice.

Why not aptly or reprepro?

The obvious move is to reach for a repository manager and let it worry about the metadata. I tried. aptly keeps a database of exactly this information, so adding a package should be a small delta rather than a full rebuild.

The catch is what happens at publish time. On every publish aptly stats and links every package file in its own local pool, so it needs the whole pool present on disk each run. In a job that starts from nothing, that means downloading the entire pool every time, which is the exact cost I was trying to remove. You can keep the pool on a persistent disk to avoid that, but then I am back to running and paying for standing storage, and the “just a bucket and a job” design is gone.

So for this shape, a stateless tool plus a cache I control is both simpler and cheaper. The whole state I have to keep is one file: the Packages I published last time.

The incremental indexer

First the job fetches what it needs: the current pool listing as relative paths, and the new .deb files on disk. The listing is one bodiless call, normalised to the same pool/... form the Filename field uses. The download then pulls only the paths the cached Packages has never seen, keeping their pool layout so dpkg-scanpackages emits the right Filename:

1
2
3
4
5
6
7
8
9
10
11
12
13

# what the pool holds today, as pool/... paths, not gs:// URLs
gcloud storage ls -r "gs://example-apt-repo/pool/stable/**/*.deb" \
  | sed "s|gs://example-apt-repo/||" > pool.list

# fetch only the paths the cache does not already describe
grep '^Filename: ' Packages.cache | awk '{print $2}' | sort > cached.list
comm -23 <(sort pool.list) cached.list | while read -r path; do
  mkdir -p "$(dirname "$path")"
  gcloud storage cp "gs://example-apt-repo/$path" "$path"
done

# scan just those new debs; the rest of the pool is not on disk
dpkg-scanpackages -m -a amd64 pool/stable > Packages.new

That scan is the only place a .deb is ever read, and it only ever sees the new ones.

Now the splice, the one part with any real logic. It parses the previous Packages into stanzas keyed by their Filename, then walks the current listing: reuse the cached stanza where we have one, use a freshly scanned stanza where we do not. Anything no longer in the pool falls out, because we only emit paths that are in the current listing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

def parse(path):
    """Packages file -> {filename: stanza}."""
    stanzas = {}
    try:
        text = open(path).read()
    except FileNotFoundError:
        return stanzas
    for block in filter(str.strip, text.split("\n\n")):
        filename = next(
            line.split(": ", 1)[1]
            for line in block.splitlines()
            if line.startswith("Filename: ")
        )
        stanzas[filename] = block.strip()
    return stanzas

cache = parse("Packages.cache")     # last run's output
fresh = parse("Packages.new")       # scan of only the new .debs
current = set(open("pool.list").read().split())  # every pool path, from gcloud storage ls

merged = {}
for path in current:
    merged[path] = fresh.get(path) or cache[path]   # scanned if new, cached otherwise

body = "\n\n".join(merged[k] for k in sorted(merged)) + "\n"
open("Packages", "w").write(body)

The Packages.cache comes from the bucket at the start of the run, and the finished Packages goes back at the end. apt-ftparchive release builds Release from the Packages files, not the packages themselves, so it stays cheap no matter how large the pool grows, and gpg signs it with the repository key ($KEYID, imported from Secret Manager at the start of the run):

1
2
3
4
5
6

apt-ftparchive release dists/stable > dists/stable/Release

gpg --batch --yes --default-key "$KEYID" \
  --clearsign -o dists/stable/InRelease dists/stable/Release
gpg --batch --yes --default-key "$KEYID" \
  --armor --detach-sign -o dists/stable/Release.gpg dists/stable/Release

Finally, push dists/ back with caching disabled so no client ever reads a stale index:

1
2
3

gcloud storage rsync -r -c \
  --cache-control="no-cache, no-store, must-revalidate" \
  dists gs://example-apt-repo/dists

Real repositories serve more than one architecture, so in practice the scan and splice run once per architecture (amd64, arm64, and so on), each writing its own Packages under binary-<arch>. The logic does not change, only the -a flag and the output path do.

Is the output actually identical?

An incremental index that is subtly wrong is worse than a slow one: it fails on the user’s machine, later, with the “Hash Sum mismatch” from earlier, and nobody can install anything.

So I built a pool of three packages and generated the index the incremental way, reusing a cached Packages and scanning only the one new .deb. Then I generated it the authoritative way, a full scan of all three.

A diff of the two is the whole test, and they match exactly, with only one of the three packages ever opened:

1
2
3

$ diff Packages.full Packages.incremental && echo IDENTICAL
IDENTICAL
reused 2 cached, scanned 1 new, total 3

Same bytes, a third of the reads. That ratio is the point, and it widens with the repo: the work stays one new package read whether the pool holds fifty packages or fifty thousand.

Running it with nothing kept warm

The indexer is a shell script and a couple of tools, so the container is small: a Debian base with dpkg-dev, apt-utils, gnupg, and the Cloud SDK. It runs as a Cloud Run job, not a service. A job is the right option here: it has no HTTP endpoint and nothing to keep listening. It starts, reindexes, and exits, and you are billed for those seconds and nothing else.

The job pulls the signing key from Secret Manager at the start of each run and imports it into a throwaway keyring:

1
2

gcloud secrets versions access latest --secret=apt-signing-key \
  | gpg --batch --import

Give the job a dedicated service account with object read and write on the repository bucket and access to the secret. Nothing else.

Hardening: keep the signing key out of the container

There is one point that could be considered a weakness in this design, especially in a highly secure environment. gcloud secrets versions access pulls the private signing key out of Secret Manager and into the container’s memory on every run. This is the key that vouches for every package every client installs, so a compromise of the indexer is a compromise of the key and from there a potential attack on everyone downstream.

For a hobby repository that is a fair trade for simplicity. For anything people actually depend on, the key should never leave a place built to hold it.

Cloud KMS is that place. You generate the key inside KMS, it is non-exportable, and the job never sees it. The indexer stops asking for the key and starts asking KMS to produce a signature, which is a much smaller thing to trust the job with.

1
2
3
4
5
6
7
8
9
10
11
12
13

gcloud kms keyrings create apt --location=europe-west2

gcloud kms keys create release-signing \
  --keyring=apt --location=europe-west2 \
  --purpose=asymmetric-signing \
  --default-algorithm=rsa-sign-pkcs1-4096-sha256 \
  --protection-level=hsm

# the indexer's service account may sign, nothing more
gcloud kms keys add-iam-policy-binding release-signing \
  --keyring=apt --location=europe-west2 \
  --member=serviceAccount:example-apt-indexer@PROJECT.iam.gserviceaccount.com \
  --role=roles/cloudkms.signerVerifier

The hsm protection level keeps the private key in a FIPS 140-2 Level 3 hardware module.

Here is the wrinkle to be honest about. apt wants an OpenPGP signature, and KMS only does raw asymmetric signing: it hands back a bare RSA signature, not the OpenPGP-framed one apt verifies. The bridge is a smartcard shim. Google ships a PKCS#11 library, libkmsp11, that exposes the KMS key through the standard smartcard interface, and gnupg-pkcs11-scd lets gpg treat that as a card. gpg then builds an ordinary OpenPGP signature while the private-key operation happens inside KMS.

Point libkmsp11 at the key ring and tell it to generate a throwaway certificate for the key, because gnupg-pkcs11-scd discovers keys by their certificate:

1
2
3
4

# kmsp11.yaml
tokens:
  - key_ring: "projects/PROJECT/locations/europe-west2/keyRings/apt"
generate_certs: true

Inside the Cloud Run job the attached service account is already the credential, so libkmsp11 authenticates with no key file. You point it at that config and wire gpg to the shim:

1
2
3
4
5
6
7
8
9
10
11
12

export KMS_PKCS11_CONFIG=/etc/kmsp11.yaml

# ~/.gnupg/gpg-agent.conf
scdaemon-program /usr/bin/gnupg-pkcs11-scd

# ~/.gnupg/gnupg-pkcs11-scd.conf
providers kms
provider-kms-library /usr/lib/libkmsp11.so

# ~/.gnupg/gpg.conf   (match the key's digest, or KMS rejects the signature)
cert-digest-algo SHA256
digest-algo SHA256

That digest line matters more than it looks. The key is an rsa-sign-pkcs1-4096-sha256 key, so KMS will only sign a SHA-256 digest. Let gpg fall back to its SHA-512 default and every signature bounces.

Then, once, you wrap the KMS key as an OpenPGP key so gpg has something to sign with:

1
2
3

gpg --card-status                     # gpg-agent picks up the KMS key
gpg-connect-agent 'SCD LEARN' /bye    # prints the keygrip
gpg --expert --full-generate-key      # choose (13) Existing key, paste the keygrip

The result lists as sec>, which is gpg’s way of saying the secret lives on a card, and here the card is KMS. Export its public key, publish it next to the repository as before, and the signing commands from earlier do not change at all. Only where the key lives does.

I ran this end to end against a real KMS key before writing it down. The InRelease it produces verifies as a good signature in a clean keyring holding only the exported public key, which is all an apt client ever does.

Two things caught me out, so they are worth a mention here:

  1. The key has to be RSA. The PKCS#11 path is dependable for RSA and fiddly for elliptic curves, so the ed25519 key from earlier becomes an RSA 4096 one.
  2. If you test against a software-protection key rather than an HSM one, add allow_software_keys: true to the config, because libkmsp11 skips software keys by default.

It is more machinery in the signing path than a plain download, but for a key that other people’s trust depends on, one that never leaves the vault is the right trade.

Triggering the reindex

Something has to run the job when a new package lands. The simplest thing that works and what I do, is to make it the last step of my CI/CD that already publishes the package.

My release pipeline uploads the .deb to the bucket, then runs the job:

1

gcloud run jobs execute apt-repo-indexer --region=europe-west2 --async

--async fires it and moves on. Swap in --wait if you want the pipeline to block until the repository is updated. That is the whole trigger. The pipeline already has credentials and already knows a release happened, so there is nothing to subscribe to and no event to keep alive.

You could instead drive it from the bucket, with an Eventarc trigger on object changes, so anything landing in pool/ reindexes on its own. It is a tidy idea and I sketched it, but for a single publisher it is more moving parts than calling the job from the thing that just did the upload.

The one case to think about is concurrency. If two releases land within a few seconds of each other, two executions can run at once and race to rewrite the same dists/. If that is a real risk for you, have the job take a small lock at the start: write a lock object with a timestamp, and if a recent one already exists, exit early so a burst of releases collapses into a single reindex. At a low release cadence you can skip it and let the last run win.

Using it

On the client, trust the public key and add the source. The modern, non-deprecated way keeps the key in its own file and points at it explicitly, rather than piping into apt-key:

1
2
3
4
5
6
7

curl -fsSL https://apt.example.com/public-key.gpg \
  | sudo tee /usr/share/keyrings/example.gpg > /dev/null

echo "deb [signed-by=/usr/share/keyrings/example.gpg] https://apt.example.com stable main" \
  | sudo tee /etc/apt/sources.list.d/example.list

sudo apt update && sudo apt install example-agent

The signed-by line is what ties the source to your key. It tells apt to trust this repository only when it is signed by that specific key and nothing else on the system.

What you end up with

A repository that is a bucket, a load balancer in front of it and a job that wakes up only when there is something to do.

No server to patch, no certificate to renew by hand, no compute idling overnight and a reindex whose cost tracks the change, not the archive.

Most of my career has been spent optimising workflows and refusing to redo work that has not changed, and this design is that same habit pointed at a package repo. Once the reindex only touches the packages that changed, a private apt repo stops being a service you babysit and becomes a bucket you forget about. If you build one, I would like to hear how it goes.

#gcp#apt#debian#cloud-run#gcs#packaging#serverless
← previousRun your own ChatGPT: self-hosted LLMs with Docker Compose
// related
$