Challenge 67 ☆☆

Welcome to challenge Challenge 67.

Secrets shipped to your cloud log sink

Applications rarely log a secret on purpose. What happens far more often is that an "audit event", a request dump, or a structured log field carries a credential along with it. Because the value is encoded, nobody notices it during code review: it just looks like an opaque blob.

In this challenge the application writes such an audit event to standard out. It never returns the secret through an endpoint and never mounts it as a file. The only place the value shows up is the log stream that your cloud provider collects for you, so you will have to go and query the log sink of the cloud you are running on.

Note that the logged value is Base64 encoded, so finding the line is only half of the work.

Tip: this is not the same as challenge 8. There the answer is logged in plain text and you can read it straight from your local container logs.

💡 Tip: Secrets are often strings, numbers, or encoded values. Copy and paste exactly what you find.

You can solve this challenge by the following steps:

  1. Find the log group of the WrongSecrets container in CloudWatch Logs. With the EKS setup from the aws folder the logs of the pod are shipped to a log group named after the cluster, for example /aws/containerinsights/<clustername>/application.

  2. Query the log group for the audit event, for instance with CloudWatch Logs Insights:

    aws logs start-query \
  --log-group-name "/aws/containerinsights/<clustername>/application" \
  --start-time $(($(date +%s) - 3600)) \
  --end-time $(date +%s) \
  --query-string 'fields @message | filter @message like /encoded credential/'

    Note: in this example the application generated the secret in the last hour, so if it is up and running longer: select a longer start-time!

  3. Alternatively tail it directly: aws logs tail "/aws/containerinsights/<clustername>/application" --follow --filter-pattern "encoded credential".

  4. Take the Base64 blob from the message and decode it: echo '<blob>' | base64 -d. That decoded value is the answer.

Not seeing anything yet? The event is emitted the first time the challenge is opened, so hit the page once and query again.

Why logging a secret to your cloud provider is a problem

Encoding is not encryption. Base64 keeps a credential out of sight during a quick code review, but anybody who can read the log sink can decode it in one command. Treat an encoded secret in a log line as a plain text secret.

A few things that make this particularly nasty in a cloud setup:

  • the audience is much wider than you think. Log sinks are usually readable by the whole platform or SRE team, and often by any workload with a broad logs:FilterLogEvents, roles/logging.viewer or Log Analytics reader permission. The blast radius of the secret becomes the blast radius of your logging permissions.

  • retention outlives rotation. Logs are commonly kept for months and replicated into an archive bucket or a SIEM. Rotating the credential does not remove the old value from those copies, so you have to treat every downstream sink as compromised too.

  • structured logging makes it easy to leak by accident. Attaching a whole payload, request or MDC context to an "audit event" is convenient, and it pulls in whatever happens to be in that context, including tokens and keys.

  • logging can move the secret into additional storage systems, archives, or third-party observability platforms. This is usually not covered by the threat model you wrote for that secret.

What to do instead:

  • never put credentials in a log statement, not even encoded, and not even at DEBUG.

  • redact at the source. Filter sensitive keys before they reach the appender, for example with a Logback converter or a masking layout, so a future code change cannot reintroduce the leak.

  • scan for it. Secret detection tooling can run against log output as well as against source code.

  • if it did happen: rotate the secret, then clean up or expire every sink and archive that received it.


⬅️ Previous
🏠 Main Page
Next ➡️
0