AWS Secrets Manager

AWS Secrets Manager stores, rotates, and retrieves secrets — database credentials, API keys, OAuth tokens — via a managed service with IAM-governed access and KMS encryption. It removes the need to check secrets into code or configuration, and makes credential rotation an automated, auditable process.

Key Features:

Automatic Rotation: Built-in rotation for RDS, Aurora, Redshift, and DocumentDB using Lambda rotation functions; custom rotation for any secret via your own Lambda.
Versioning with AWSCURRENT / AWSPREVIOUS / AWSPENDING: Zero-downtime rotation — clients always fetch the current version, applications can roll back to previous if needed.
KMS Encryption: At-rest encryption with AWS-managed or customer-managed KMS keys; cross-account access via resource policies on the CMK.
Fine-Grained IAM: Per-secret resource policies, tag-based access, and VPC endpoint policies for private retrieval.
Replication: Multi-region replicas for DR and locality.
Audit: Every GetSecretValue is logged to CloudTrail for access auditing.

Secrets Manager vs. Parameter Store:

Systems Manager Parameter Store is free for standard parameters, supports SecureString with KMS, but has no built-in rotation and is limited to 8 KB for advanced parameters.
Secrets Manager costs ~$0.40/secret/month + API calls but adds rotation, cross-region replication, and richer resource policies.
Rule of thumb: credentials -> Secrets Manager; config values -> Parameter Store.

Common Patterns:

Lambda / ECS / EKS Retrieval: Pull secrets at startup via IAM role; cache in-memory for the task lifetime.
RDS Master Password Rotation: Attach a secret to the RDS instance and enable managed rotation.
Third-Party API Keys: Store provider keys (Stripe, Datadog, OpenAI) with custom rotation Lambdas.
Cross-Account Access: Producer account stores secret; consumer accounts retrieve via resource policy and shared KMS key.
Sidecar Caching: AWS Secrets Manager Lambda Extension or Secrets Manager CSI driver in EKS caches secret values close to the workload.

Service Limits & Quotas:

Secrets per region: default soft limit 500,000 (raisable).
Secret value size: 64 KB max (hard).
API throughput: starts at 5,000 GetSecretValue/sec and 50 administrative calls/sec per account per region (soft).
Versions per secret: 100 (older versions auto-pruned).
Rotation frequency: minimum every 4 hours, maximum 1,000 days.
Recovery window: 7-30 days after deletion before permanent removal (or force delete).

Pricing Model:

$0.40 per secret per month (prorated by hour).
$0.05 per 10,000 API calls.
Multi-region replicas: billed as a separate $0.40/month secret in each replica region.
Common cost surprise: applications calling GetSecretValue on every request instead of caching — at high QPS this dwarfs the per-secret cost. Use the AWS-provided caching client or Secrets Manager Lambda Extension.
Free tier: 30-day trial per new secret, then standard pricing.

Code Example — Retrieve and Cache a Secret:


import boto3, json
from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = boto3.client("secretsmanager", region_name="us-west-2")
cache = SecretCache(
    config=SecretCacheConfig(secret_refresh_interval=3600),
    client=client,
)

def db_creds():
    raw = cache.get_secret_string("prod/app/db")
    return json.loads(raw)
    # {"username": "app", "password": "...", "host": "...", "port": 5432}

# First call hits Secrets Manager; subsequent calls return cached value.
creds = db_creds()

Create a Secret with Rotation (CLI):


aws secretsmanager create-secret \
  --name prod/app/db \
  --description "Postgres app user" \
  --secret-string '{"username":"app","password":"REPLACE-ME"}' \
  --kms-key-id alias/secrets

aws secretsmanager rotate-secret \
  --secret-id prod/app/db \
  --rotation-lambda-arn arn:aws:lambda:us-west-2:111122223333:function:RotateRDSPostgres \
  --rotation-rules AutomaticallyAfterDays=30

Common Interview Questions:

How does zero-downtime rotation work?

The rotation Lambda runs four phases: createSecret generates a new password and stages it under the AWSPENDING label; setSecret updates the database with the new credential; testSecret verifies it works; finishSecret moves the AWSCURRENT label to the new version and demotes the previous one to AWSPREVIOUS. Applications always request AWSCURRENT and rollback is one label move away.

When would you pick Parameter Store over Secrets Manager?

Non-sensitive config (feature flags, regional endpoints), or sensitive values where you don't need rotation and the cost of $0.40/secret/month at scale matters. Parameter Store standard tier is free; advanced is $0.05/parameter/month.

How do you safely share a secret across AWS accounts?

Use a customer-managed KMS key (AWS-managed keys can't grant external access), attach a resource policy to the secret allowing the consumer account principal, and grant the consumer's KMS key policy decrypt rights. The consumer then calls GetSecretValue across accounts.

What's the right way to load secrets in a high-throughput Lambda?

Fetch outside the handler so the value is reused across warm invocations, and either set a TTL refresh or use the Secrets Manager Lambda Extension that runs as a local HTTP cache. Never call GetSecretValue on every request — it adds latency and dominates cost.

Can you delete a secret immediately?

Only with --force-delete-without-recovery, which bypasses the 7-30 day recovery window. Default deletion is recoverable so a typo or compromised admin can't permanently destroy production credentials.

What happens if rotation fails halfway through?

The AWSPENDING label remains attached to the staged version but AWSCURRENT is unchanged, so applications keep working with the old credential. CloudWatch alarms on the rotation Lambda errors are essential — silent rotation failures can let credentials expire if the database has independent password expiry.