Metadata-Version: 2.4
Name: aac-trust-anchor-publisher
Version: 0.2.3
Summary: AAC tenant-side trust-anchor publisher daemon: pushes root-key JWK Sets to the AAC control plane.
Author: Agent Authority Cloud Project
License-Expression: Apache-2.0
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cryptography>=42.0
Requires-Dist: pyjwt>=2.8
Requires-Dist: httpx>=0.28
Provides-Extra: test
Requires-Dist: pytest>=8.0; extra == "test"
Dynamic: license-file

# aac-trust-anchor-publisher

The trust-anchor publisher is a small daemon that runs inside your
organization and keeps the AAC control plane's copy of your public trust
material current. It reads public keys and CA certificates from directories
you control, packages them, signs the upload with your tenant-admin key, and
pushes the result to the control plane at startup and whenever a file
changes. Counterparties never contact the publisher: their sidecars fetch
what it published from the control plane's public endpoints.

It plays up to two roles, and you configure only the roles your tenant
needs:

* **root-keys**, for a tenant whose workloads mint authority chains. The
  publisher packages the root signing **public** keys as a JWK Set and
  publishes them so verifiers can check your chain roots.
* **spiffe-bundle**, for a tenant whose workloads present certificates. The
  publisher packages the CA certificates for one SPIFFE trust domain into a
  trust bundle and publishes it so receivers can verify your workload
  identities.

Run one publisher per tenant for root keys, and one per trust domain for
bundles (see "One tenant, two trust domains" below). Each publisher is the
single writer for what it publishes.

## What the publisher holds, and what it never sees

* It holds the **tenant-admin private key**, the credential that
  authenticates uploads to the control plane. This is the only private key
  the daemon touches. Its public half is registered for your tenant when
  the tenant is registered (`aac tenant register --tenant-admin-pubkey-file`
  names it); until that registration exists every upload is rejected with
  `ERR_TENANT_ADMIN_KEY_UNKNOWN`.
* It reads **public** material only: root signing public keys and CA
  certificates. Root signing private keys stay with the workloads that mint
  chains and never pass through the publisher.
* The control plane never receives any private key of yours.

Keep the private halves apart: mount the tenant-admin private key and the
public directories read-only into the publisher, and mount root signing
private keys only into the sidecars that mint. When you retire an
environment, revoke the public root key, stop the publisher, and remove its
tenant-admin key under your retention procedure. The control plane keeps
tenant-admin key history; there is no separate revocation call for it.

## Install

Pick the channel that matches your host. Both run the same daemon.

**Container** (Kubernetes, Docker, or any orchestrator):

```bash
docker pull ghcr.io/cascadeauth/aac-trust-anchor-publisher:0.2.3
```

The image is multi-architecture (`linux/amd64` and `linux/arm64`). Pull by
version, not `latest`, so a redeploy runs what you tested.

**pip** (a Linux host you manage, where systemd runs the daemon):

```bash
sudo python3 -m venv /opt/aac/venv
sudo /opt/aac/venv/bin/pip install aac-trust-anchor-publisher
/opt/aac/venv/bin/aac-trust-anchor-publisher
```

Use a dedicated virtual environment: distribution Pythons refuse
system-wide pip installs. The wheel installs the
`aac-trust-anchor-publisher` command. Run with no configuration, as in the
last line, it exits with code 2 and names `AAC_TAP_TENANT_ID` as missing.
That is the expected result and proves the command is installed.

## Configuration

The daemon is configured entirely through environment variables. It reads
them once at startup and refuses to start on a mistake: a missing required
variable, a half-configured role, zero roles, or an unreadable admin key
prints the reason and exits with code 2. Restarting does not fix that class
of error; fix the configuration. An unreachable or refusing control plane
is not fatal: the daemon keeps polling and retries.

Always required:

| Variable | Meaning |
| --- | --- |
| `AAC_TAP_TENANT_ID` | Your tenant id in its canonical form, `tnt-` followed by a UUID, for example `tnt-550e8400-e29b-41d4-9716-446655440000`. Not a trust domain. |
| `AAC_TAP_ADMIN_KEY_FILE` | Path to the tenant-admin Ed25519 private key, an unencrypted PKCS#8 PEM file that begins `-----BEGIN PRIVATE KEY-----`. |

Root-keys role. Set both or neither; setting one is a startup error:

| Variable | Meaning |
| --- | --- |
| `AAC_TAP_ROOT_KEYS_DIR` | Directory of root signing public keys, one `<key_id>.pub.pem` file each. |
| `AAC_TAP_ROOT_KEYS_INGEST_URL` | Full root-keys ingest URL, for example `https://api.example/v1/root-keys/ingest`. |

SPIFFE-bundle role. Set the first three together; the fourth is optional:

| Variable | Meaning |
| --- | --- |
| `AAC_TAP_SPIFFE_BUNDLE_DIR` | Directory of CA certificates, one `<anchor_id>.ca.pem` file each. |
| `AAC_TAP_SPIFFE_BUNDLE_INGEST_URL` | Full SPIFFE-bundle ingest URL, for example `https://api.example/v1/spiffe-bundle/ingest`. |
| `AAC_TAP_SPIFFE_TRUST_DOMAIN` | The trust domain this bundle is authoritative for: your assigned hosted domain, or a custom domain you have proven and bound, for example `tenant.example`. Not the tenant id. |
| `AAC_TAP_SPIFFE_BUNDLE_READ_URL` | Optional. The full public URL where the control plane serves this trust domain's bundle, for example `https://trust.example/.well-known/spiffe-bundle/tenant.example`. Set it when public reads use a different host from ingest. An absolute `http(s)` URL with a path and no credentials, query or fragment. |

Optional, with defaults:

| Variable | Default | Meaning |
| --- | --- | --- |
| `AAC_TAP_POLL_INTERVAL_SECONDS` | 60 | How often each role re-reads its directory. The production floor is 30; a lower value is a startup error unless `AAC_TAP_DEV_ALLOW_FAST_POLL` is set. |
| `AAC_TAP_REQUEST_TIMEOUT_SECONDS` | 10 | Timeout for each HTTP request to the control plane. |
| `AAC_TAP_LOG_LEVEL` | info | `debug`, `info`, `warning` or `error`. |
| `AAC_TAP_BACKOFF_CAP_SECONDS` | 900, or the poll interval when that is larger | Ceiling for the wait after repeated failed uploads. Must not be below the poll interval. |
| `AAC_TAP_DEV_ALLOW_FAST_POLL` | unset | `1` or `true` permits a poll interval below 30 seconds. For test harnesses only; never set it in production. |

Your onboarding material names the ingest URLs and, when it differs, the
public trust host. Both are used over HTTPS with certificate verification;
plain HTTP works only for local test deployments.

After a failed upload the daemon waits longer before the next attempt: the
wait grows exponentially with random jitter, never past
`AAC_TAP_BACKOFF_CAP_SECONDS`, and resets after the next accepted upload or
unchanged poll.

## Publishing root keys

### Root-key directory

`AAC_TAP_ROOT_KEYS_DIR` holds one `<key_id>.pub.pem` file per root signing
public key. The whole filename stem is the key id (dots are allowed): the
JWK your counterparties see carries it as `kid`, and every chain your
workloads mint names it, so choose ids you can rotate by, such as
`root-2026-q3`. Keys are Ed25519 or EC P-256. Files that do not end in
`.pub.pem` are ignored, and a file that fails to parse is skipped with a
warning and re-checked at every poll.

Never place a private key in this directory. The daemon would not publish
it, but it does not belong on the publisher host.

An empty directory is not published: the daemon logs `no publishable keys`
and waits for files to appear.

### Rotating root signing keys

Rotation is a transition set: add the new `<key_id>.pub.pem` beside the old
one, wait one poll interval and confirm an `ingest accepted` line, roll your
minting workloads to the new private key, then delete the old file. The
control plane serves a removed key as `DEPRECATED` for 24 hours so chains
signed just before the change still verify. Keep at least one key from the
current set while you transition: a set that drops every known key at once
is refused.

## Publishing a SPIFFE bundle

### CA directory

`AAC_TAP_SPIFFE_BUNDLE_DIR` holds one `<anchor_id>.ca.pem` file per CA
certificate; the stem is the anchor id. Each certificate must parse, carry
`BasicConstraints CA=TRUE`, and be unexpired; a file that fails is skipped
with a warning. The bundle covers every workload identity under the
configured trust domain, `spiffe://<trust_domain>/*`.

CA rotation is whole-document: add the new CA beside the old one (one
bundle carries both), wait for the next upload, then delete the old file
once every workload certificate has rolled. The bundle replaces atomically;
the overlap lives in the document, not in a server-side grace period.

At every poll that finds CA files, the daemon reads the binding state for
the trust domain from the public read URL (or the URL it derives from the
ingest URL) and uploads only while the binding is `ACTIVE`, at the binding's
current version. A `503` from the read URL while the binding is `ACTIVE` is normal
before the first bundle exists. A revoked binding, a trust domain with no
binding, or a read response that names a different trust domain stops
publication with a diagnostic naming the cause; the daemon never falls back
to another host.

### Separate ingest and trust hosts

When the control plane serves public reads from a different host than
signed uploads, set all three:

```bash
AAC_TAP_SPIFFE_TRUST_DOMAIN=tenant.example
AAC_TAP_SPIFFE_BUNDLE_INGEST_URL=https://api.example/v1/spiffe-bundle/ingest
AAC_TAP_SPIFFE_BUNDLE_READ_URL=https://trust.example/.well-known/spiffe-bundle/tenant.example
```

The read URL must be the public bundle URL for exactly the configured trust
domain. The daemon sends plain `GET` requests there and signed `POST`
requests only to the ingest URL. When the read URL is unset, the daemon
derives it from the ingest URL: it removes `/v1/spiffe-bundle/ingest` (or,
for any other ingest path, keeps only the scheme and host) and appends
`/.well-known/spiffe-bundle/<trust_domain>`.
Root-key-only publishers need no read URL.

### One tenant, two trust domains

A tenant that runs workloads under its assigned hosted domain and under a
custom domain runs two publisher processes, one per trust domain, with
separate directories. Only one of them publishes root keys.

| Setting | Publisher for the hosted domain | Publisher for the custom domain |
| --- | --- | --- |
| `AAC_TAP_TENANT_ID` | Same tenant id | Same tenant id |
| `AAC_TAP_SPIFFE_TRUST_DOMAIN` | Hosted domain | Custom domain |
| `AAC_TAP_SPIFFE_BUNDLE_DIR` | Hosted CA anchors | Custom CA anchors |
| `AAC_TAP_SPIFFE_BUNDLE_READ_URL` | Hosted domain's read URL | Custom domain's read URL |
| Root-keys role | Configured, if the tenant mints | Unset |

Adding a custom domain does not retire the hosted one. Workloads move to
the custom domain by receiving new identities, not by renaming existing
ones.

## Rotating the tenant-admin key

The tenant-admin key signs every upload. The daemon reads it once at
startup, and the control plane activates a replacement key the instant it
is registered, with no grace period: the old key stops verifying at once.
Rotation is therefore a short, deliberate sequence, and it differs between
the two deployment paths.

Generate the new pair anywhere safe; only the public half leaves the host
you will run on:

```bash
openssl genpkey -algorithm ed25519 -out tenant-admin.new.pem
openssl pkey -in tenant-admin.new.pem -pubout -out tenant-admin.new.public.pem
```

**pip and systemd:** stage the new private key, stop the service, register
the public half with the AAC CLI, point the environment file at the new
key, and start the service:

```bash
sudo install -m 640 -o root -g aac tenant-admin.new.pem /etc/aac/keys/tenant-admin.new.pem
sudo systemctl stop aac-trust-anchor-publisher
aac tenant rotate-admin-key --tenant-id tnt-550e8400-e29b-41d4-9716-446655440000 \
  --tenant-admin-pubkey-file tenant-admin.new.public.pem
sudo sed -i 's|^AAC_TAP_ADMIN_KEY_FILE=.*|AAC_TAP_ADMIN_KEY_FILE=/etc/aac/keys/tenant-admin.new.pem|' \
  /etc/aac/trust-anchor-publisher.env
sudo systemctl start aac-trust-anchor-publisher
```

**Container:** a container's environment and mounts are fixed when it is
created, so overwrite the key at the existing bind-mount source and
recreate the container. Do not repoint `AAC_TAP_ADMIN_KEY_FILE`, and do not
rely on `docker start`, which can keep the old file:

```bash
docker stop aac-trust-anchor-publisher
aac tenant rotate-admin-key --tenant-id tnt-550e8400-e29b-41d4-9716-446655440000 \
  --tenant-admin-pubkey-file tenant-admin.new.public.pem
sudo install -m 640 -o root -g aac tenant-admin.new.pem /etc/aac/keys/tenant-admin.pem
docker rm -f aac-trust-anchor-publisher
```

Then repeat the `docker run` command from "Running in a container"
unchanged; the paths it names now hold the new key.

Either way, the first upload after the restart proves the rotation: watch
for `ingest accepted`. A rejection with `ERR_TENANT_ADMIN_KEY_UNKNOWN` or
`ERR_TENANT_ADMIN_SIGNATURE_INVALID` means the daemon came back on the old
key.

If the key is compromised, rotate first to kill the exposed key, then
replace the file and restart as fast as practical. Only new uploads pause
while the publisher is down; anchors already published keep serving, and
agent traffic does not depend on this key.

## Running under systemd

The daemon is a plain foreground process (no forking, no pid file, logs to
stdout and stderr) and lets systemd own its lifecycle. Create `/etc/aac/keys`
and `/etc/aac/root-keys`, copy the tenant-admin private key to
`/etc/aac/keys/tenant-admin.pem` and your `<key_id>.pub.pem` files into
`/etc/aac/root-keys/` first; the complete unit file ships inside the wheel,
so the rest is a short sequence:

```bash
sudo useradd --system --home /var/lib/aac aac
sudo tee /etc/aac/trust-anchor-publisher.env > /dev/null << 'EOF'
AAC_TAP_TENANT_ID=tnt-550e8400-e29b-41d4-9716-446655440000
AAC_TAP_ADMIN_KEY_FILE=/etc/aac/keys/tenant-admin.pem
AAC_TAP_ROOT_KEYS_DIR=/etc/aac/root-keys
AAC_TAP_ROOT_KEYS_INGEST_URL=https://api.example/v1/root-keys/ingest
EOF
sudo chmod 600 /etc/aac/trust-anchor-publisher.env
sudo chown -R root:aac /etc/aac/keys /etc/aac/root-keys
sudo chmod 750 /etc/aac/keys /etc/aac/root-keys
sudo chmod 640 /etc/aac/keys/tenant-admin.pem
sudo find /etc/aac/root-keys -name '*.pub.pem' -exec chmod 640 {} +
/opt/aac/venv/bin/aac-trust-anchor-publisher --print-systemd-unit | sudo tee /etc/systemd/system/aac-trust-anchor-publisher.service > /dev/null
sudo systemctl daemon-reload
sudo systemctl enable --now aac-trust-anchor-publisher
journalctl -u aac-trust-anchor-publisher -f
```

systemd reads the environment file as root, so it stays `root:root` with
mode 600. The daemon itself, running as the `aac` user, opens the admin
key and every public key file, so those are `root:aac` with mode 640 in
directories the group can traverse (the `find` form enumerates the public
keys under `sudo`, because your own shell cannot list a directory it can no
longer read). A root-only admin key exits 2 at startup. A root-only public
key is skipped with a warning while the remaining readable keys are still
uploaded; nothing is published only when no readable key is left. Add the
SPIFFE-bundle variables to the same file to enable that role.

The unit sets `Restart=on-failure` with `RestartPreventExitStatus=2`, so a
configuration error stays down and visible instead of restart-looping. It
runs as the dedicated `aac` user and sandboxes the process read-only
(`ProtectSystem=strict`); the daemon never writes to the filesystem. If your
virtual environment is not at `/opt/aac/venv`, edit `ExecStart`. Logs go to
the journal only: `journalctl -u aac-trust-anchor-publisher` is the complete
history.

## Running in a container

Place the tenant-admin private key and the `<key_id>.pub.pem` files on the
host first (`/etc/aac/keys/tenant-admin.pem` and `/etc/aac/root-keys/` below),
then bind-mount them read-only and pass the configuration as environment
variables. The daemon needs outbound HTTPS to the control plane and no
inbound ports. The image runs as a non-root `aac`
user whose ids do not match your host's, so join the container to the host
group that owns the key material instead of loosening file modes:

```bash
docker run -d --name aac-trust-anchor-publisher \
  --restart unless-stopped \
  --group-add "$(getent group aac | cut -d: -f3)" \
  -e AAC_TAP_TENANT_ID=tnt-550e8400-e29b-41d4-9716-446655440000 \
  -e AAC_TAP_ADMIN_KEY_FILE=/keys/tenant-admin.pem \
  -e AAC_TAP_ROOT_KEYS_DIR=/root-keys \
  -e AAC_TAP_ROOT_KEYS_INGEST_URL=https://api.example/v1/root-keys/ingest \
  -v /etc/aac/keys/tenant-admin.pem:/keys/tenant-admin.pem:ro \
  -v /etc/aac/root-keys:/root-keys:ro \
  ghcr.io/cascadeauth/aac-trust-anchor-publisher:0.2.3
docker logs -f aac-trust-anchor-publisher
```

On the host, the key material is `root:aac` with mode 640 and the
directories mode 750 (`groupadd --system aac` first if the group does not
exist). Every host path in a `-v` option must exist before the first run:
Docker creates a missing path as a directory, and the daemon then fails
with `cannot read admin key file ... Is a directory`. Prefer `--restart
unless-stopped` over `--rm`: a container that exits 2 at startup keeps its
logs. A container stuck in `Restarting` is the container face of exit 2:
read `docker logs`, fix the configuration, and recreate.

## What the log tells you

One line per upload attempt, per role:

| Log line | Meaning | What to do |
| --- | --- | --- |
| `root-keys ingest accepted: tenant_id=… sequence=… keys=1 admin_kid=tadm_…` | The control plane accepted the upload. | Nothing. Confirm at the public endpoint below. |
| `ingest POST failed (will retry next poll): …` | The control plane was unreachable: DNS, routing, TLS or a refused connection. The detail names the layer. | Reproduce with `curl` from the daemon's own network position (inside the container for a container). |
| `ingest REJECTED (will retry next poll): status=… body=…` | The network is fine; the control plane refused the upload, and the body names the error. | `ERR_TENANT_ADMIN_KEY_UNKNOWN` or `ERR_TENANT_ADMIN_SIGNATURE_INVALID`: the registered admin public key does not match your private key. `ERR_TENANT_NOT_FOUND`: the tenant id is not registered. `ERR_INGEST_SEQUENCE_STALE`: the host clock stepped backwards; publication resumes when it catches up. |
| `spiffe-bundle publish blocked: …` | The trust domain's binding is missing, revoked or not active, or the read URL names another domain. | Bind or reactivate the domain with the AAC CLI, or fix `AAC_TAP_SPIFFE_TRUST_DOMAIN` and `AAC_TAP_SPIFFE_BUNDLE_READ_URL`. |
| `no publishable keys …` or `no publishable trust anchors …` | The directory is empty or every file was skipped. | Check the mount and the `scan skipped` warnings above it. |
| `publish failed N time(s) in a row; backing off …` | Consecutive failures; the wait grows up to the cap. | Fix the cause. A fix on the control-plane side takes effect at the next attempt; a changed admin key or environment variable takes effect only after a restart (or a recreate, in a container). |

Every upload succeeds or fails as a whole, and the daemon retries at the
next poll, so an unreachable or refusing control plane needs no restart.
Changes on the publisher host are different: the admin key and the
environment are read once at startup, so a replaced key file or an edited
variable takes effect only after you restart the service or recreate the
container, as in the rotation section above. The proof of a successful
publish is the control plane serving your material from the same public
endpoints your counterparties read:

```bash
curl -fsS https://trust.example/.well-known/aac-root-keys/tnt-550e8400-e29b-41d4-9716-446655440000
curl -fsS https://trust.example/.well-known/spiffe-bundle/tenant.example
```

Root keys appear as `{"keys": [...]}`: one entry with `aac_key_status`
`ACTIVE` per file in your directory, plus any key you removed within the
last 24 hours, served as `DEPRECATED` until its grace period ends. Seeing
the old key as `DEPRECATED` after a rotation is the expected result, not a
stale publish.
