Deployment secrets flow

Application secrets — the env vars and credentials a running container reads — live in AWS Secrets Manager, in eu-central-1, and you edit them in the AWS console. A changed value does not reach a running pod on its own: every procedure below ends by rolling the pods, and is not finished until you have. How it works is at the end, for when something looks wrong.

Getting access

Sign in at https://d-9967264414.awsapps.com/start, pick the account for your environment, and pick the twpro-secrets role — not admin or readonly, which are separate sessions with different permissions.

Environments	Sign in to	Account
dev4, dev5, devtw	Treatwell X UALA Development	194455872934
staging	Treatwell Saas Staging	056007870958
prod, prod-it	Treatwell Saas Production	388786193518

!!!warning Switch the region to eu-central-1 Sessions land in eu-west-1, where the secrets list is empty and looks like missing access. !!!

Production is read and write, so one developer can change a production value with no review. CloudTrail PutSecretValue records who changed what and when, and never captures the value.

Some buttons on the secret's page will fail, by design:

Button	Fails because
Revert to this version	needs UpdateSecretVersionStage, which the role does not have
Delete secret, Store a new secret	creating and deleting are a twpro-infra PR
Edit tags	the tw:k8s-namespace tag decides who may read the secret, so IaC owns it

Changing a value

1. Sign in with the twpro-secrets role — see Getting access.
2. Open Secrets Manager and switch the region to eu-central-1.
3. Search twpro/ and open the secret.
4. Retrieve secret value → Edit → the Key/value tab → change the value → Save.
5. Find which namespaces mount it: search k8s-gitops-twpro for the full secret name. Every secret-provider-class.yaml that matches is a namespace that mounts it, and the path names the cluster and namespace — …/overlays/prod-pita-be-main-env-prod/….
6. Restart the Deployments that consume it — see Restarting after a change.

!!!danger Saving is not the end of the job envFrom: secretRef is read when a pod starts. Once you have saved, the new value is in Secrets Manager and every running pod still holds the old one, indefinitely. A change that stops at the console has not happened. !!!

Use the Key/value tab, not Plaintext. Key/value rewrites the whole JSON document around your edit; Plaintext hands you the raw document, where deleting a key by accident is silent.

Restarting after a change

What has to restart is the Deployments that consume the secret. The k8s-gitops-twpro search tells you which namespaces they are in; inside a namespace, assume everything consumes it except web-lb, which carries HAPROXY_STATS_AUTH and nothing else.

In ArgoCD — open the Application for the namespace, find each Deployment in the resource tree, then ⋮ → Restart.

Or redeploy the environment, which rolls every Deployment in it and so covers the consumers without picking them out. This is the practical route for a shared/ secret, which is consumed across the environment rather than in one namespace. What makes or breaks it:

• It must produce a different image tag. Dev and staging images are tagged git-<sha>, so redeploying the same commit changes no manifest and rolls nothing. Push a commit first, or cut a new release for production.
• ./deploy-to-env.sh {dev4|dev5|devtw|staging} deploys the branch you are on, force-pushed. Check you are on the branch that environment should be running before using it.

For production that is a manual release — see trigger deployments when there are no code changes.

!!!warning A shared/ secret is consumed outside the namespace you were thinking of twpro/<env>/shared/<name> is mounted across the environment. Read the search result rather than restarting be-main and assuming you are done. !!!

Sync and Refresh in ArgoCD do not help: no manifest changed, so there is nothing to reconcile and no new pod. Only a restart or a genuinely new image tag makes one. CronJobs need nothing — each scheduled run is a new pod already.

Rolling back a change

1. Open the secret and its Versions tab, which lists AWSCURRENT and AWSPREVIOUS.
2. Retrieve the AWSPREVIOUS value and save it as a normal Key/value edit. ("Revert to this version" is denied to this role, so this is the route.)
3. Search k8s-gitops-twpro for the secret name to find the namespaces, as when changing a value.
4. Roll the pods — Restarting after a change.

A rollback is a value change, so it reaches running pods exactly as slowly as the mistake did.

Adding a secret, or a key

A new key in a secret that already exists:

1. Set the value in the console, as in Changing a value.
2. Add the key to that namespace's SecretProviderClass in k8s-gitops-twpro and let ArgoCD sync it. Copy the shape from a sibling entry in the same file — the alias is always <secret-name>__<KEY>.
3. Search k8s-gitops-twpro for the secret name, then roll the pods — Restarting after a change.

The value goes in first. The other order leaves jmesPath naming a key that does not exist, and the pod never starts.

A new secret needs a PR against twpro-infra, because nobody holds CreateSecret. Add the name to config/<env>.yaml under the namespace that should read it, or under shared for one every namespace in the environment reads:

be-main-env-dev4:
  - backend-something-secrets

/config/ is owned by @treatwell/pro, so it does not wait on DevX to merge — but DevX applies the stack, and the namespace you list it under becomes the secret's tw:k8s-namespace tag, which is what authorises reads.

Set the value after the apply — a secret with no version answers GetSecretValue with ResourceNotFoundException — then add it to the SecretProviderClass and to the envFrom of the Deployments that need it. That last step changes the Deployments, so ArgoCD rolls the pods itself; nothing to restart by hand.

Removing a secret, or a key

The reverse order, for the same reason: nothing may still name something that has stopped existing.

A key: remove it from the SecretProviderClass first and let ArgoCD sync, then remove it from the secret in the console. Then roll the pods, or they keep the old variable until they next restart.

A whole secret: remove it from the SecretProviderClass and from every envFrom that names the synced Secret — a Deployment naming a Secret that no longer exists does not start. That is a manifest change, so ArgoCD rolls the pods itself. Then delete the name from config/<env>.yaml in twpro-infra and have DevX apply.

!!!danger Secrets Manager holds the only copy of the value Nothing is in git any more, so a deletion cannot be recovered from a PR. The apply schedules the secret for deletion 30 days out rather than removing it at once. Inside that window DevX can bring it back — the role has no RestoreSecret, so it has to be them:

aws secretsmanager restore-secret --secret-id twpro/<env>/<namespace>/<name> --region eu-central-1

Re-adding the name to config/<env>.yaml instead does not work: a create against a name pending deletion fails with InvalidRequestException. Restore it, do not re-add it. !!!

When a mount fails

Symptom	Cause
driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers	the pod landed on a new node before the driver DaemonSet did; it resolves itself
ResourceNotFoundException	the secret is declared in config/<env>.yaml but has no value set
AccessDenied	the secret is listed under a different namespace in config/<env>.yaml, so its tw:k8s-namespace tag does not match the pod's. The role also needs DescribeSecret next to GetSecretValue; with only GetSecretValue the mount fails with an AccessDenied that reads like a tagging problem
the provider cannot resolve credentials at all	the namespace has no Pod Identity association — it is missing from local.namespaces in tofu/pod-identity/<cluster>/main.tf
the mount fails right after a SecretProviderClass change	jmesPath names a key that is not in the secret — the value has to exist before the class references it

Pod Identity credentials cache for six hours, and editing an association does not invalidate them. Recreate the pods before concluding an edit did not work.

How it works

Pods fetch secrets with the Secrets Store CSI driver, as the twpro-app ServiceAccount through EKS Pod Identity. One role per cluster, TwProSecretsReader, serves every namespace: what separates them is a session tag carrying the pod's namespace, matched against the secret's tw:k8s-namespace tag. No secret value is stored in any repository, and nothing decrypts anything in CI.

What	Where	Who owns it
the value	AWS Secrets Manager, eu-central-1	whoever holds the twpro-secrets permission set
which secrets exist	twpro-infra config/<env>.yaml	Pro, with DevX co-owning
which keys a namespace mounts	k8s-gitops-twpro SecretProviderClass	DevX and Pro
the IAM that permits the read	twpro-infra tofu/pod-identity/<cluster>/	DevX

Names are twpro/<env>/<namespace>/<name> for a secret only one namespace may read, and twpro/<env>/shared/<name> for one every namespace in the environment reads.

Application containers mount the twpro-secrets SecretProviderClass — the same name as the permission set, and unrelated to it — which carries the namespace's own secrets plus the environment's shared/ ones. A namespace that also runs a web-lb carries a second class, twpro-secrets-lb, holding HAPROXY_STATS_AUTH alone — so an edge proxy never gets the database and Redis credentials on disk.

!!!warning The synced Secrets exist only while a pod mounts the class A class writes one Kubernetes Secret per source secret, named after it — not one object named after the class. Each is owned by every ReplicaSet, and for the pods a CronJob creates every Job, whose pods mount that class. Owners are appended rather than replaced, so they survive drain, OOM and crash-loops, but scaling a namespace's workloads to zero removes the last owner and the objects go with it. They come back when a pod does. !!!