In short
Most engineering teams confuse monitoring with observability. Monitoring is about tracking known failure modes through predefined metrics and thresholds—it tells you that your CPU is at 95% or that a service is down. Observability, however, is the ability to understand the internal state of a system by looking at its external outputs (logs, metrics, traces) even when you didn't predict the specific failure beforehand. For teams running complex distributed systems on Kubernetes, relying solely on monitoring leads to 'dashboard blindness,' where every light is green but users are still reporting errors. To build truly resilient infrastructure, you need high-cardinality data that allows you to pivot from a single error spike to the specific customer ID or container version causing it.
The Monitoring Trap
In the early days of web infrastructure, monitoring was straightforward. You had a handful of monolithic servers, and you cared about four things: CPU usage, memory availability, disk space, and network throughput. If these stayed within certain bounds, your application worked. This is what we call 'known-unknowns.' You know that high memory usage might cause an OOM (Out Of Memory) error, so you set a threshold at 85% and wait for the alert.
However, as we moved toward microservices and container orchestration via Kubernetes, the failure modes changed. In a distributed environment, systems don't just fail by crashing; they fail through subtle degradations. A single slow database query in one service might cause a connection pool exhaustion in another, which then triggers a cascading timeout in a third. If you are only monitoring 'service uptime,' all your dashboards will show green because every individual pod is technically running and responding to health checks.
The symptom vs the cause
Monitoring identifies symptoms (the service is slow). Observability identifies causes (the latency is specific to requests coming from a particular API gateway version during a specific deployment window).
When an incident occurs in a complex pipeline, the first question shouldn't be 'Is it up?' but rather 'What is happening right now that we didn't expect?' If your telemetry doesn't allow you to ask that question, you aren't practicing observability; you are just watching a very expensive thermometer.
The Three Pillars and High Cardinality
To move from monitoring to observability, we rely on the three pillars of telemetry: Metrics, Logs, and Traces. While these terms are often used interchangeably in marketing fluff, their engineering roles are distinct and must be integrated to provide value.
- Metrics: Aggregated numerical data over time. They are incredibly efficient for alerting (e.g., Prometheus scraping an endpoint) but lack context. A metric can tell you that error rates rose by 5%, but it won't tell you which user was affected.
- Logs: Discrete events recorded by the application. Logs provide deep context but are expensive to store and difficult to query at scale if they aren't structured properly.
- Traces: The glue that connects them. Distributed tracing (using OpenTelemetry) follows a single request as it hops through multiple services, providing a timeline of exactly where latency was introduced.
The real secret to observability is cardinality. Cardinality refers to the number of unique values in a dataset. A metric like 'request_count' has low cardinality. A metric like 'request_count{user_id="12345", container_id="abc-xyz"}' has extremely high cardinality. In a traditional monitoring setup, adding these dimensions would crash your database or explode your costs. In an observability-first architecture, you design your telemetry so that you can slice and dice by any dimension—region, version, customer tier, or even specific feature flags—without pre-defining the queries.
# Example of a high-cardinality trace attribute in OpenTelemetry
span.set_attribute("app.customer_tier", "enterprise")
span.set_attribute("app.deployment_id", "v2.4.1-build88")
span.set_attribute("app.request_path", "/api/v1/orders")Implementing the Stack
Building an observability pipeline is not about buying a single tool; it's about creating a data flow. We typically recommend an architecture centered around OpenTelemetry (OTel). OTel provides a vendor-neutral way to collect and export telemetry, preventing you from being locked into a specific provider like Datadog or New Relic.
A standard production implementation looks like this:
- Instrumentation: Using OTel SDKs within your code (Python, Go, Node.js) to automatically capture spans and metrics.
- The Collector: Running an OpenTelemetry Collector as a sidecar in Kubernetes or as a standalone service. This collector receives data, processes it (e.g., stripping PII), and exports it to multiple backends.
- Storage & Visualization: Sending metrics to Prometheus/Grafana for real-time alerting, logs to an ELK stack or Loki for deep searching, and traces to Tempo or Jaeger for request visualization.
This decoupled approach is vital. If you decide to switch your backend from a self-hosted Grafana instance to a managed SaaS solution, you only change the configuration in your Collector, not your application code. This is especially important when managing complex API integration layers where latency can hide in the network hops between third-party services and your internal logic.
The Gotchas: Where Teams Fail
Observability is not a silver bullet. If implemented poorly, it can become a massive cost center or even degrade the performance of your application. We have seen several common pitfalls in recent deployments.
1. The Cost Explosion
High cardinality is powerful, but every unique label you add to a metric increases the memory footprint of your time-series database. If you start putting 'session_id' into a Prometheus metric, your server will run out of RAM within hours.
2. The Logging Verbosity Trap
Teams often try to solve observability issues by simply turning on 'DEBUG' logging across the entire cluster. This creates a massive amount of noise, makes searching difficult, and can significantly increase your cloud storage bill without providing actionable insights.
3. Ignoring Traces
Many teams invest heavily in metrics but treat tracing as an afterthought. Without traces, you are essentially looking at a map of the city (metrics) without knowing which specific car is stuck in traffic (traces). In microservice architectures, logs alone cannot reconstruct the journey of a single failed request.
To avoid these, we suggest a tiered approach: use metrics for broad alerting, structured logs for local context, and traces for cross-service debugging. Never treat them as interchangeable substitutes.
The Outcome: Reduced MTTR
When you move from a monitoring mindset to an observability mindset, the primary metric that improves is Mean Time To Resolution (MTTR). In traditional setups, engineers spend 80% of an incident performing 'detective work'—trying to correlate logs with metrics and manually checking server statuses. With high-cardinality telemetry, that detective work is automated.
Imagine a scenario where your checkout service starts failing intermittently. A monitoring alert tells you the error rate has spiked. Instead of logging into five different servers to check tailing logs, an engineer opens a trace for one of the failed requests. They immediately see that the request passed through the gateway correctly but stalled at a specific database call because of a lock contention caused by a background batch job. The root cause is identified in minutes rather than hours.
This level of clarity is essential when managing custom software development projects where the business logic is unique and cannot be solved by looking up common error codes online. You are building your own patterns, so you need your own visibility into those patterns.
Lessons Learned & Next Steps
If we were to design an observability strategy from scratch today, we would focus on three things: standardization, sampling, and developer experience. First, standardize everything on OpenTelemetry immediately. Do not let different teams pick different libraries; it makes correlation impossible.
Second, implement intelligent sampling for traces. You don't need to record 100% of successful requests—that is a waste of money. Instead, capture 100% of errors and only 5% of successful requests. This gives you the data you need for debugging without the massive storage overhead.
Finally, make observability part of your deployment pipeline. If a new version of a service doesn't include the necessary telemetry attributes to be debuggable, it shouldn't pass through CI/CD. Observability is not something you add after the code is written; it is a core requirement of the software itself.
For teams looking to scale their infrastructure or optimize their current DevOps practices, we recommend starting with a small audit of your existing telemetry. Are you actually able to answer 'why' when things break? If not, it might be time to move beyond simple dashboards and start building a real observability platform.