Observability That Answers Real Questions, Not Just Dashboards
Most projects arrive at observability backward: logs first because they're free with console.log, metrics next because a dashboard feels reassuring, and tracing last, if ever. We used to build the same way, and it meant that when a specific customer complained checkout was slow, we had graphs showing p50 and p95 latency looking healthy and no way to explain their individual experience.
We now treat traces as the primary tool for debugging any specific incident and metrics as the tool for spotting trends. Adding OpenTelemetry instrumentation across a client's checkout service — API, payment provider call, inventory check, order write — took about three days, and the first week it was live we found that a specific combination of promo code plus split shipment triggered a redundant inventory re-check adding 800ms, invisible in aggregate p95 numbers because it affected under 2% of orders.
Logs still matter, but we got strict about structure. Unstructured log lines like 'processing order' are close to useless at scale; we moved every service to structured JSON logs with a consistent trace_id field so a single request can be reconstructed across API, worker, and database logs in our log aggregator (we use Grafana Loki, chosen mainly for cost compared to hosted alternatives at the client's log volume).
The metric we found most useful wasn't a technical one. We added a handful of business-level metrics — successful checkouts per minute, payment failures by provider, cart abandonment rate — alongside the usual CPU and latency graphs. A payment provider outage shows up in 'payment failures by provider' within seconds, well before it would show up as elevated latency, since a fast failure doesn't move the latency needle at all.
The lesson that stuck: instrument for the question you'll actually ask during an incident ('why was this request slow'), not the question that's easiest to graph ('what is average latency'). Averages hide the customer who's actually having a bad time.
