Secrets should live with their failure domain
Centralizing every secret can simplify the diagram while making recovery, authority, and outages harder to reason about.
“One secret manager” sounds like a clean platform rule. It gives every credential one inventory, one policy language, and one place to look.
It also gives every workload a dependency on the same provider, network path, and recovery authority.
RS Platform started with a centralized secret store. As its boundaries became more concrete, the better rule emerged: keep references centralized in Git, but keep secret values with the infrastructure that must recover them.
A reference is safe to share
Repositories contain names, ownership, expected consumers, and delivery contracts. They do not contain values.
That distinction lets Git describe the whole secret topology without becoming the secret store. A review can answer:
- Which service expects this value?
- Which runtime identity may read it?
- How does rotation reach the consumer?
- What evidence proves the new version is active?
- Which recovery actor can restore it?
The value itself stays out of commits, Terraform state, workflow output, and shell history.
Provider-local authority reduces strange dependencies
AWS gateway keys belong in AWS Secrets Manager because the AWS recovery path must work even when another cloud is unavailable. Cluster and home-workload values belong in an AWS Parameter Store lane protected by a dedicated KMS key because that is where their current consumers and operators live.
When GKE was part of the design, GCP workloads used GCP-local secret delivery. Moving the runtime back to AWS removed that lane rather than preserving a central store for symmetry.
This is deliberate duplication of mechanism, not duplication of values. Each failure domain owns the secrets needed to reconstitute itself.
Terraform should own the envelope
Infrastructure code creates KMS keys, IAM policies, paths, audit controls, and consumer identities. It does not declare secret values or placeholder resources merely to claim ownership.
An operator-only command performs writes through a narrow, short-lived role. It accepts protected input, rejects paths outside approved prefixes, and returns metadata rather than the value. Bootstrap, rotation, and rollback use the same boundary.
This keeps a Terraform plan from becoming a secret transport and keeps provider state from accumulating plaintext by accident.
Rotation is not complete at write time
A new version in a secret manager proves only that storage changed. It does not prove that a workload reloaded it, that a replacement node received it, or that the old issuer still accepts a rollback value.
Every consumer needs a refresh contract and observable evidence. Some can reload; some need a rollout; some credentials require overlap between issuer versions. The rotation procedure belongs to the consuming system, not just to the backing store.
Optimize for recovery clarity
The cleanest secret diagram is not necessarily the safest one. A useful design lets an operator point at any failed domain and answer two questions:
- Which authority can rebuild it?
- Which other domains must still be alive?
If recovering an AWS gateway requires GCP, a CI system, and a human laptop that cached the right file, centralization has hidden coupling rather than removed it.