tech, developers, and the code underneath

issue 043· essay·

Your logs are a product and you are shipping a bad one

Structured events, one line per unit of work, and the specific reason your grep-based debugging is slow.

Here is a log line from a real production system, lightly anonymized:

2025-06-02 14:23:11 INFO  Processing request

Processing which request? For whom? Started when? Finished when? Did it succeed? This line costs money to produce, store, index, and retain, and it conveys approximately zero bits of information.

Multiply by four hundred million a day.

the mental model that fixes this#

Stop thinking of logs as a narrative of what your program did. Think of them as structured events about units of work, emitted once, with everything you know attached.

The canonical form: one event per request, per job, per message consumed. Wide. Structured. Emitted at the end when you know how it went.

json
{
  "event": "http_request",
  "trace_id": "4bf92f3577b34da6",
  "method": "POST",
  "route": "/api/orders",
  "status": 201,
  "duration_ms": 143,
  "user_id": "u_8812",
  "org_id": "org_44",
  "db_queries": 7,
  "db_ms": 89,
  "cache_hits": 3,
  "cache_misses": 1,
  "upstream_ms": 22,
  "region": "us-east-1",
  "version": "2025.06.02-a1b2c3d",
  "feature_flags": ["new_checkout", "fast_path"]
}

One line. Now you can answer questions you did not think to ask when you wrote it:

  • Which org has the slowest p99 on this route?
  • Are requests with new_checkout enabled making more database queries?
  • Did latency change after the deploy of a1b2c3d?
  • Is the cache miss rate correlated with the slow requests, or is that coincidence?

None of those are answerable by grepping "Processing request."

the rules#

Log at the boundary of a unit of work, not at every step inside it. Interior logging is what tracing is for. If you need step-level visibility, emit spans, not log lines.

Never log a string you have to parse later. "user " + id + " failed" means somebody writes a regex. Put id in a field.

Attach identity to everything. Trace ID, user, org, request ID. The single most common debugging failure is having the information and being unable to correlate it.

High cardinality is the point. The advice to avoid high-cardinality fields comes from metrics systems, where cardinality multiplies storage. Events are not metrics. user_id in an event is exactly what makes it useful, and any observability tool that cannot handle it is the wrong tool.

Log the decision, not the branch. Not "entering fast path". Rather "fast_path": true on the one event.

Errors get context, not just stack traces. A stack trace tells you where. The fields tell you which input, which user, which version, which flag combination. Where is the easy part.

what this costs#

Fewer lines, each larger. In practice, storage usually goes down, because you delete the interior noise. Query performance improves dramatically because you are filtering structured fields rather than doing full-text search.

The real cost is discipline at write time, and one afternoon to set up a logger that makes the structured path the easy path. If emitting a field requires more typing than emitting a string, people will emit strings.

the test#

Pick an incident from the last quarter. Ask: could I have diagnosed that from logs alone, without adding new logging and redeploying?

If the answer is no — and it usually is — that is the gap. The information you needed was in the process's memory at the time and you did not write it down.

Write it down.

Dom, June 3, 2025

get README in your inbox

One dispatch, no noise. Tech and developer news, plus the occasional long piece on the craft.

subscribe →