Signing pipeline

Architecture reference for the create -> approve -> sign -> deliver pipeline, for anyone reading or changing the server’s code. It consolidates the six implementation phase documents that built the pipeline; those are gone, but every decision that still constrains the code is recorded here.

Status: built and working for all four certificate types. User, PAM and console certificates go through the signer; service certificates go through enrollment and are signed when a code is redeemed. server/service/pipeline_test.go drives the user path with a real Router, real gochannel transport, real signer, and in-memory sqlite, and asserts a CA-verifiable certificate comes out the far end.

Transport is Watermill’s in-memory gochannel in the default single-process mode. The split into separate signer and API processes is implemented: with pubsub.backend set to nats, the same pipeline runs across ssoosshd serve api and ssoosshd sign processes – see startup modes.

Why it looks like this

Wait/notifyWaiter used to be an in-process map[string]chan struct{}, which could not survive a restart or scale past one process. Separately, signing needed a home, and the decision was to keep it out of the webserver’s memory space entirely rather than implement it inline – the CA private key should not sit alongside HTTP handling and OIDC/LDAP calls.

Why message-passing rather than RPC

A direct gRPC/mTLS call from the approving node to the signer would have to either block that node’s thread waiting for the result, or re-publish the result to whichever node holds the SSE connection – two hops instead of one, with a blocked thread in between.

With Watermill the signer publishes the result directly to wherever the SSE thread is listening. One signaling model end to end, and the cluster stays stateless with respect to “which node owns this request”: no node is pinned to a request from approval through delivery.

Components

Three logical components, one process:

Component	Package	Database	Holds CA key
API / webserver	server/controller, server/service	yes	no
Signer	server/signer	no	yes
Listener / resolver	server/service/signreply.go	yes	no

The signer’s zero-database property is enforced by a test: server/signer/zerodb_test.go runs go list -deps and fails if gorm.io/gorm or server/service enters its dependency graph. That test is the only thing keeping the boundary honest – it must stay green.

server/certmsg exists because of that boundary: the signer cannot import server/service (which pulls gorm, and would become an import cycle once the listener landed in service). It holds the wire shapes and topic names – SigningJob, SignedReply, SignQueueTopic, SignedTopic, WaitTopic, RequestedOptions – with no dependencies of its own. service.RequestedOptions is an alias of certmsg.RequestedOptions, so there is one type rather than two kept in sync by hand.

Topics

Topic	Shape	Durability need
certrequest.wait.<id>	one per request	none – the database is truth
certrequest.sign	shared queue	none – at-most-once by decision, the human retries (see Decisions)
certrequest.signed	shared reply topic	none – same decision

The signed-reply topic is shared rather than per-request because only the listener ever subscribes to it – there is no fan-out concern.

How the processes and NATS fit together

The split topology: a serve api process (HTTP handlers plus the listener), a sign process, NATS carrying the signing subjects (certrequest.sign, certrequest.signed) and the CA key subjects (ca.key.announce, ca.key.request) between them, and the database on the api side only. Numbers trace one request.

flowchart LR
    accTitle: How the processes and NATS subjects fit together
    accDescr {
      Ten numbered steps. The ssoossh client posts a request to serve api and
      waits on a server-sent events stream. A browser approves. serve api
      publishes a signing job on the NATS subject certrequest.sign; the sign
      process, which holds the CA key and has no database, consumes it and
      publishes the signed certificate on certrequest.signed. serve api writes
      the audit row to PostgreSQL, then wakes the waiting request on
      certrequest.wait.id, which fans out to whichever instance is holding the
      client’s stream, and the certificate is delivered over it.
    }
    C["ssoossh client"]
    C -->|"1: POST + wait on SSE"| API

    API["serve api<br/>HTTP + listener"]
    W["browser"] -->|"2: approve"| API

    subgraph N["NATS"]
        SIGN(["certrequest.sign"])
        SIGNED(["certrequest.signed"])
        WAIT(["certrequest.wait.id"])
    end

    API -->|"3: signing job"| SIGN
    SIGN -->|"4"| S["sign<br/>CA key, no database"]
    S -->|"5: signed certificate"| SIGNED
    SIGNED -->|"6"| API
    API -->|"7: audit row"| DB[("PostgreSQL")]
    API -->|"8: wake + certificate"| WAIT
    WAIT -->|"9: fan-out"| API
    API -->|"10: deliver over SSE"| C

Steps 8 and 9 look like the api process talking to itself, and in this two-process picture it is – but the hop through the wait topic is what makes scaling out transparent. Run several serve api instances behind a load balancer and nothing in the diagram changes: certrequest.sign is a queue group (“signer”), so exactly one signer takes each job; certrequest.signed is a queue group (“signed-listeners”), so exactly one instance’s listener records the audit row and publishes the wake; certrequest.wait.<id> has no queue group, so it fans out and reaches whichever instance holds the client’s SSE stream – even when that is not the instance that took the approval. No instance is pinned to a request at any point.

In the default single-process mode the picture is the same minus the middleman: the identical subjects ride Watermill’s in-memory gochannel instead of NATS, and API, listener, and signer are goroutines in one process.

Flow

1. Client POSTs a request, and gets events_url + approval_url.
2. Client opens events_url (SSE), so Wait subscribes to certrequest.wait.<id>.
3. A browser opens approval_url, authenticates, approves. Approve does not sign. It resolves policy, persists the row as signing, publishes a self-contained job to certrequest.sign, and returns immediately.
4. The signer consumes the job, signs, publishes to certrequest.signed.
5. The listener writes the audit row, delivers the certificate, marks the request terminal.
6. The client’s Wait returns, and eventsHandler sends the terminal SSE event.

sequenceDiagram
    accTitle: The request lifecycle across api, signer and listener
    accDescr {
      The client posts a request and gets back an events URL and an approval
      URL, then opens the events stream, which subscribes a waiter. An
      approver authenticates and approves in a browser. serve api resolves
      policy and moves the row to signing, then publishes the job on
      certrequest.sign. The signer signs with the CA key and replies on
      certrequest.signed. The listener writes the audit row, caches the
      certificate and marks the request terminal, then wakes the waiter on
      certrequest.wait.id, and serve api sends the certificate down the
      client’s stream.
    }
    autonumber
    participant Client as ssoossh client
    participant API as serve api
    actor Approver as browser
    participant Signer as signer
    participant Listener as listener

    Client->>API: POST request
    API-->>Client: events_url + approval_url
    Client->>API: open SSE, Wait subscribes
    Approver->>API: authenticate and approve
    API->>API: resolve policy, row = signing
    API-)Signer: job [certrequest.sign]
    Signer->>Signer: sign with CA key
    Signer-)Listener: reply [certrequest.signed]
    Listener->>Listener: audit row, then cache + wake
    Listener->>Listener: mark request terminal
    Listener-)API: wake [certrequest.wait.id]
    API-->>Client: certificate over SSE

See Interactive user certificates for the user-facing sequence diagrams of the same path.

Decisions that still constrain the code

Certificates are ephemeral

Signed certificates are never persisted. They are delivered exactly once through the in-memory resolved cache and its wake message; a client that misses the window re-requests, which is cheap because they are short-lived by design.

The consequence is load-bearing: Wait can find a terminal approved row with nothing cached. It must not report success with an empty certificate – it returns errorresponses.CertificateUnavailableError (410 Gone). The 410 is deliberate: the client reconnects only after a stream it had already established drops, never after a status the server answered with, so the client stops rather than reconnect-loops.

Note the asymmetry: an enrollment token is durable (a column on the request row), so enrolled is rebuilt from the database rather than failing.

Listener ordering is load-bearing

resolveSuccess does audit row -> deliver (cache + wake) -> status update, in that order.

Delivering before the status update is what makes “approved with a cold cache” mean only “the process restarted” – exactly the case the 410 describes. Written the other way round, a live server would briefly show approved with nothing cached and wrongly tell clients their certificate was gone.

The cost is that a crash between delivery and the status update leaves the row in signing. That is what the sweep exists to clean up, and the audit row is already durable by then.

Persistent: false on gochannel

Originally set true. It is both unnecessary and actively wrong:

• Unnecessary – the wake topic’s race is already covered by Wait’s DB fallback (see below).
• Wrong – gochannel replays a Persistent topic’s entire history to every new subscriber, not just missed messages. A signer restart would re-receive every job ever queued, already-signed ones included.

It also never evicts, so Persistent meant unbounded memory growth. Full reasoning lives on the gochannel.Config literal in server/pubsub.New.

Wait subscribes before reading the database

Not because of message replay (that mechanism is gone), but because Wait re-reads the database immediately after subscribing, and reconcileStatus’s terminal branch re-notifies whenever it finds a resolved-but-not-yet-cached status. A publish that races ahead of the Subscribe call is caught one loop iteration later via the database, not lost.

notifyWaiter treats a failed publish as non-fatal. The database write it follows already succeeded and is the durable fact; a lost wake message is a latency problem a reconnect resolves, not a correctness one.

Nacks busy-loop, so retries live in middleware

gochannel redelivers a Nack immediately, with no backoff (watermill@v1.5.3 /pubsub/gochannel/pubsub.go:412), so “return an error to retry” is a CPU spin, not a retry policy.

pubsub.New installs middleware.Retry (backoff) wrapped in a dropAfterRetries middleware that gives up and acks, logging the discarded payload. A dead-letter topic is deferred until there is a durable broker to put one on.

This is why the sweep must be periodic – see below.

Ack semantics: what is real, what is not

Achieved: correct ordering. The signer publishes its reply before acking; the listener audits and delivers before updating status. A signing failure is a successful handle (ack); only transport failures nack.

Not achieved: at-least-once redelivery across a restart. gochannel is in-memory and non-persistent, so a crash loses in-flight jobs outright regardless of ack timing. NATS core is at-most-once too, and JetStream is declined by design: the flow is short and interactive, so the person waiting is the retry mechanism (see Decisions).

Router lifecycle

Router.Run(ctx) does not return when ctx is canceled – only an explicit Router.Close() unblocks it. PubSub.Run watches ctx.Done() and calls Close() itself.

Router.Close() then waits up to CloseTimeout (default 30s) for handlers, and PubSub.Run could call it before Router.Run had started them. PubSub.Run now waits on Router.Running() first, and CloseTimeout is set to 3s so it cannot overrun bootstrap’s 5s shutdown budget.

What Approve actually decides

Approve resolves policy and hands off; it never signs.

Field	Behavior
Extensions	Requested, intersected with the type’s configured ceiling, then with the lifetime policy’s grant set, minus its remove set (narrowRequestedOptions, then policyOutcome.narrowOptions)
ValidDuration	The type’s ceiling, narrowed by the lifetime policy’s tiers and source rules; it only ever shortens
ForceCommand	Dropped from the request – no config concept to bound it against
SourceAddresses	Dropped from the request; a source rule may pin a service certificate to the address the request came from
NoTouchRequired	Only ever granted where the type allows it (service certificates)
Who may approve	The type’s require conditions. require_group is retired, and setting it is a startup error rather than a silently ignored key
Principals	The approver’s selection, validated against the accounts they hold (username plus other_accounts); with none, the username for a user request and every held account for PAM or console. A service certificate takes the selected service account, which must be one the approver holds

Dropping a client-requested ForceCommand or SourceAddresses is fail-closed, not an oversight: granting an unbounded critical option would violate “server config is the outer bound on every option.”

Type routing

• User, PAM and console – through the queue (flowSigning, via approveForSigning): Approve publishes a signing job and the listener delivers the result.
• Service – bypasses the queue (flowEnrollment). There is no signer round trip to wait on, because the certificate isn’t produced until a later service retrieve redeems the code. Approve creates an enrollments row holding the code, the public key, and the option set, key ID, principals and lifetimes fixed at approval, marks the request enrolled, and calls notifyWaiter synchronously. Redemption signs from that row and never re-derives policy.

The stranded-request sweep

CertRequestService.SweepStrandedRequests (server/service/sweep.go), scheduled from server/bootstrap/scheduler.go.

Policy: invalidate, don’t resume. Replaying isn’t safe to reason about (did the signer already sign and die before publishing the reply?), and re-requesting is cheap.

It must be periodic, not boot-only

A healthy, never-restarted process can strand a request:

1. dropAfterRetries deliberately drops a message once retries are exhausted – the alternative on gochannel is an infinite busy-loop. A dropped job leaves the row in signing with nothing left to resolve it.
2. reconcileStatus applies no TTL to signing – that branch just says “keep waiting.” The approval TTL only bounds how long something stays pending.

Deriving the cutoff without a new column

Nothing records when a request entered signing. The bound is derived from created_at instead:

stranded if:  status = 'signing'
              AND created_at < now - (ApprovalTTL + SigningGrace)

A request is approved somewhere between created_at and created_at + ApprovalTTL, so the latest it can have entered signing is created_at + ApprovalTTL. This can never invalidate a healthy in-flight request.

Accepted imprecision: a request approved immediately waits ApprovalTTL + SigningGrace to be swept rather than SigningGrace. Erring long is the right direction – the cost is a client waiting longer for bad news, whereas erring short would cancel certificates about to be issued.

Both terms are shares of one configured budget: ApprovalTTL and SigningGrace derive from cert_options.client_timeout, which must be positive, so there is no configuration in which the formula degenerates and no special case for an unbounded approval window. The sweep also runs on the SigningGrace interval, which is why the worst-case client wait spends two of those shares and why ApprovalTTL reserves for both – the three add up to exactly client_timeout.

A bulk UPDATE is not sufficient

Wait only re-reads the database when a message arrives on its wake topic. A direct SQL update publishes nothing, so a client blocked on a swept request keeps blocking even though its row now reads failed.

The sweep selects stranded rows and, per row, updates and calls notifyWaiter. Per-row also keeps the update guarded (WHERE id = ? AND status = 'signing'), matching Deny, expire, and the listener’s markResolved, so a concurrently-resolved request isn’t overwritten.

Gotchas worth not rediscovering

• Serial numbers are masked to 63 bits. Go’s database/sql refuses to bind a uint64 with the high bit set, so random 64-bit serials failed the audit insert roughly half the time – the certificate was never delivered and the client hung. Caught only by the end-to-end test, and only intermittently.
• Terminal status names live in internal/apitypes (TerminalStatuses()). They were once duplicated as literals in internal/api/sse.go, which is how enrolled shipped as a status the client treated as informational – every service enrollment hung. Add a status in one place only.
• AutoMigrate (tests) and the embedded SQL migrations (production) can silently diverge. Verify schema changes against the real migration, not just the test path.

Known gaps

• signing_started_at column would remove the sweep’s imprecision, which is that a request approved immediately still waits the whole approval share before being swept. Reconsider if that becomes a practical problem.
• The sweep’s age-threshold approach is safe for several instances sharing a database: every instance derives the same bound from the same configured budget. See multi-instance-safety-plan.md.

The NSS directory, beside the pipeline

The NSS directory shares no code path with signing. It is a read path: the server’s directory sync fills POSIX tables, /api/nss/* answers lookups from them under a fleet key, and a daemon on each host turns those answers into users and groups.

flowchart LR
    accTitle: The processes in an NSS lookup, and which of them is ours
    accDescr {
      A caller on the host, such as sshd or sudo, runs glibc, which loads
      nss-systemd from systemd. nss-systemd connects over varlink to the
      socket com.github.mnestor.ssoossh in /run/systemd/userdb/, which
      systemd holds for ssoossh-nss. ssoossh-nss, running as root with no
      capabilities, answers from its cache or asks ssoosshd over pinned
      HTTPS with the fleet key. ssoosshd answers from its posix tables,
      which the directory sync fills from LDAP. Only ssoossh-nss and
      ssoosshd are ssoossh code; nothing of ours runs inside the caller.
    }
    subgraph Caller["Every resolving process (not ours)"]
        G["glibc"] --> M["nss-systemd"]
    end
    M -->|"varlink, AF_UNIX<br/>/run/systemd/userdb/com.github.mnestor.ssoossh"| D["ssoossh-nss<br/>nssd/, cmd/ssoossh-nss"]
    D -->|"pinned HTTPS, fleet key"| S["ssoosshd<br/>/api/nss/*"]
    S --> T[("posix_users, posix_groups")]
    L["LDAP"] -->|"directory sync"| T

Piece	Where	What it does
nss-systemd	systemd	the NSS module glibc loads; queries every socket in /run/systemd/userdb/ in parallel and turns JSON user and group records into struct passwd and struct group
ssoossh-nss.socket	deploy/	systemd holds the userdb socket, so lookups queue across a daemon restart instead of failing
ssoossh-nss	nssd/, cmd/ssoossh-nss	implements systemd’s io.systemd.UserDatabase (GetUserRecord, GetGroupRecord, GetMemberships), org.varlink.service.GetInfo, and its own com.github.mnestor.ssoossh.NSS.RefreshUser; the cache, the host policy, the upstream client
internal/varlink	shared	the NUL-framed JSON codec, used by the daemon and by ssoossh host principals for its one-way refresh hint
internal/api	shared	the HTTPS client, with SPKI pinning and the fleet key headers
/api/nss/*	server/	the lookup endpoints, the fleet keys, and the directory sync behind them

nssd/ imports only internal/, never server/ or client/, and the daemon is Linux-only, built with CGO_ENABLED=0 and the server’s FIPS 140-3 setting.

Some properties the design rests on:

• The daemon answers fast, always. systemd’s varlink call timeout is 45 s, and a service that does not answer holds every lookup on the host that long. Every call is answered within socket_answer_deadline; a fetch still running then completes into the cache, and the caller gets ServiceNotAvailable. Over eight upstream fetches in flight, a new one is refused at once rather than queued.
• sshd is recognised by its cgroup. A uid 0 peer whose systemd cgroup is one of sshd_units gets a synchronous refresh of a stale entry, so a group removed in the directory is gone at the next login. Not comm, which any process can set, and not /proc/<pid>/exe, which an empty capability bounding set cannot read from a full-capability sshd; only root can move a process into sshd.service.
• Privileged data goes to root only. The record’s privileged section, which becomes the shadow entry, is sent only to uid 0 peers; everyone else gets the record marked incomplete.
• Local names never leave the host. A name or id in /etc/passwd or /etc/group is answered NoRecordFound without an upstream call, because nss-systemd asks every userdb service during every initgroups, local users included.

Why there is no NSS module of our own

The obvious design is a libnss_ssoossh.so.2 that glibc loads, and it was the first draft. It was dropped for systemd userdb:

• No code of ours in setuid callers. An NSS module runs inside sudo, su, passwd and sshd. With userdb that module is systemd’s, and our exposure there is the content of the records we serve, which nss-systemd validates and copies.
• No private protocol to version. The contract between caller and daemon is systemd’s documented user and group API, not one of ours.
• One process to audit, holding the key, the cache and the policy, where a module would have needed a daemon beside it anyway: unprivileged callers cannot write a shared cache, and the key cannot be readable by every process that resolves a user.

The cost is the platform floor (systemd 245, glibc, systemd in nsswitch.conf) and that “service not available” reaches callers as “not found”. Both are on the host page. The alternatives considered are in Decisions.