Secrets management, practically

Most secret management advice is a product comparison. Here is the practice instead, ordered by what to do first.

1. stop long-lived credentials existing#

The single highest-value change, and it is architectural rather than a tool.

A static credential can be stolen, leaked, committed, logged, or exfiltrated from a developer laptop. A credential that lives for fifteen minutes and is scoped to one operation cannot be usefully stolen.

Workload identity. Your CI job, your container, your function proves its identity to the cloud provider and receives a short-lived token. No stored secret at all.

# GitHub Actions with OIDC — no stored cloud credentials
permissions:
  id-token: write
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::123456789012:role/deploy
      aws-region: us-east-1

That eliminates the most commonly stolen class of credential entirely. If you do one thing from this article, do this one.

Database credentials can work the same way — IAM authentication, or dynamically generated credentials from a secret manager with a short lease.

2. keep them out of the places they end up#

Not in version control. Obvious, universally violated. Run a scanner in pre-commit and in CI. Both — pre-commit catches it before it happens, CI catches it when someone skipped the hook.

Once committed, assume compromised. Rewriting history does not help; the object is in every clone and in every fork. Rotate, then clean up.

Not in the container image. Layers are inspectable. docker history and a layer extraction tool will find it.

Not in environment variables, ideally. This is more controversial. Environment variables leak: into crash dumps, into child processes, into /proc, into logs when someone prints the environment for debugging, into error tracking services that capture context.

Files with restrictive permissions, mounted at runtime, are better. Environment variables are convenient and are the pragmatic choice for many systems — just know what you are accepting.

Not in logs. Redact at the logger, with a deny-list of key names, not at each call site. Somebody will forget at a call site. The logger never forgets.

REDACT = {"password", "token", "secret", "api_key", "authorization", "cookie"}

def scrub(d):
    return {k: ("***" if k.lower() in REDACT else v) for k, v in d.items()}

Not in error tracking. Most error trackers capture local variables and request headers by default. Configure the redaction, then verify it by triggering a test error and reading what arrived.

3. rotate, and test the rotation#

Rotation is only real if it has been executed. A rotation procedure that has never run is a document, not a control.

The property that makes rotation painless: support two valid credentials at once. Add the new one, deploy, verify, remove the old one. Without overlap, rotation requires downtime, so it does not happen.

Design for this when you create the credential, not when you need to rotate it under incident pressure.

4. scope narrowly#

A credential should permit exactly what its holder needs.

• Read-only where writes are not needed.
• One bucket, one prefix, not the account.
• One database, one schema, one set of tables.
• Time-bounded where possible.

The test: if this credential leaked, what could an attacker do? If the answer is "anything," the scope is wrong regardless of how well you protect it.

5. know what you have#

An inventory of every secret: what it is, where it lives, what it grants, who owns it, when it was last rotated.

Most organizations cannot produce this, which means they cannot answer "what do we rotate" during an incident, which turns a two-hour response into a two-day one.

the incident procedure#

When a secret is exposed, in this order:

1. Rotate first. Before investigating, before cleanup, before the postmortem. Every minute the credential is valid is a minute of exposure.
2. Then assess what it could reach, and check logs for use.
3. Then clean up the exposure.
4. Then work out how it happened.

The common mistake is investigating first. While you investigate, the credential is live.

the tooling note#

Every major cloud has a secret manager. They are all adequate. The dedicated tools add dynamic credential generation, fine-grained policy, and audit — genuinely useful at scale and not where to start.

Start with: OIDC workload identity for machine-to-machine, a cloud secret manager for what remains, scanning in CI, redaction in the logger, and a tested rotation procedure.

That covers the overwhelming majority of real-world credential compromise, and it is a week of work rather than a platform migration.