Building Systems That Tell You What Is Wrong
Monitoring tells you when something is broken. Observability tells you why, and that distinction is the foundation of the modern operations toolkit teams now rely on. Popularized by Charity Majors and others around 2017, it kicked off a rethinking of how engineers understand production systems. Traditional monitoring consisted of pre-defined dashboards and alert thresholds. Observability is the ability to ask arbitrary questions about your system's internal state from its external outputs, even questions you did not think to ask when you built the system.
The Three Pillars of Observability
Metrics: counters, gauges, and histograms that describe system state at a point in time. Request rate is 10K/second. P99 latency is 450ms. Error rate is 0.3%. Metrics are cheap to store and ideal for alerting and dashboards.
Logs: discrete events with context. '2024-01-15T09:15:22Z user_id=12345 endpoint=/checkout status=500 duration_ms=2341 error=database_timeout'. Logs are expensive to store at volume but essential for debugging specific incidents.
Traces: the journey of a single request through distributed services. This HTTP request took 800ms total: 150ms in the API gateway, 400ms waiting for the database query, 250ms generating the response. Traces are essential in microservice architectures where a single user request traverses 10 to 20 services.
Each pillar is incomplete alone. A metric alert fires: P99 latency spiked to 2 seconds. Which service? A trace shows: it is the database query in the payment service. Which query? A log entry shows: it is the 'get_account_balance' query taking 1.8 seconds when account_id is not indexed. Metrics alert. Traces localize. Logs explain.
Every system design should include observability from day one. Adding it later is an architectural retrofit. Instruments that are not built in are instruments that are not there when you need them.
Feature Flags and Progressive Delivery (2018)
The old deployment model: write code, test it, deploy to everyone, pray. If something breaks, roll back, which is itself often broken. Feature flags separate deployment (putting code in production) from release (enabling it for users). Code ships continuously. Features turn on gradually.
Progressive delivery patterns built on feature flags:
Canary deployments: enable a new feature for 1% of users. Monitor error rates, latency, and business metrics. If healthy after 30 minutes, expand to 10%, then 50%, then 100%.
Blue-green deployments: maintain two identical production environments. New version deploys to the inactive environment. Switch traffic at the load balancer level. Rollback is instantaneous by switching back.
A/B testing: different feature variants for different user segments. Measure conversion rate, engagement, or revenue impact before committing to one version.
Kill switches: a feature flag that can disable a feature instantly without a deployment. When something goes wrong at 2 AM, turning off a feature is much faster and safer than rolling back.
Dark launching: send production traffic to a new implementation in parallel with the existing one. Log and compare results. Fix discrepancies before switching traffic.
Feature flags transform deployment risk from a step function (everything works, or everything breaks) into a gradual curve. The blast radius of any change is bounded by the percentage of users with the flag enabled.
Platform Engineering (2020)
Microservices gave teams autonomy. They also created fragmentation. Each team chose different languages, containerization tools, CI/CD pipelines, and observability stacks. The cognitive overhead on developers was crushing. Being effective meant being an expert in Kubernetes, Terraform, Prometheus, and your own domain. Most engineers wanted to focus on their product. Instead they were managing infrastructure.
Platform engineering emerged as the answer: a dedicated team that builds internal developer platforms (IDPs) abstracting infrastructure complexity behind self-service workflows. Spotify open-sourced Backstage in 2020 as a service catalog and developer portal. Teams can discover existing services, deploy new ones, view runbooks, and access analytics from one interface. The platform team maintains the golden paths. Product teams follow them without needing to understand the underlying infrastructure.
The key principle: the best infrastructure is invisible to the developers using it. A developer should be able to describe 'I want a service with a PostgreSQL database, Redis cache, and Kubernetes deployment' and receive a working environment. The platform translates that intent into the actual Terraform, Kubernetes manifests, and CI/CD pipelines without the developer touching any of it.
Further Reading
Charity Majors, Liz Fong-Jones & George Miranda: 'Observability Engineering' (O'Reilly)
OpenTelemetry documentation (opentelemetry.io) for the standard instrumentation framework
LaunchDarkly's guide to feature flags for practical implementation patterns
Backstage documentation (backstage.io) for the leading internal developer platform
Team Topologies by Matthew Skelton & Manuel Pais for the organizational patterns behind platform engineering
OpenTelemetry: The Standard That Ended Vendor Lock-in
Before OpenTelemetry, every observability vendor (Datadog, New Relic, Jaeger, Zipkin) required you to instrument your code with their proprietary SDK. Switching vendors meant re-instrumenting every service. OpenTelemetry (OTel) emerged from the merger of OpenCensus (Google) and OpenTracing (CNCF) in 2019. It provides a vendor-neutral API and SDK: you instrument your code once with OTel, and configure which backend receives the data. Switch from Jaeger to Honeycomb by changing one config line, not by touching application code. The OTel collector runs as a sidecar or daemon, receives telemetry from services, and exports to any backend via its plugin system.
Auto-instrumentation: OTel agents automatically instrument common libraries (HTTP servers, database clients, message queues) with zero code changes.
Manual instrumentation: add custom spans with attributes for business-level context (user_id, order_id, feature_flag_value).
Propagation: OTel injects trace context (trace ID, span ID) into HTTP headers and message queue metadata automatically, connecting traces across service boundaries.
Sampling: recording every trace is expensive. Head-based sampling (sample X% of traces at ingestion) or tail-based sampling (sample all traces with errors or high latency) control cost.
Semantic conventions: OTel defines standard attribute names (db.system, http.method, net.peer.port). Consistent naming enables cross-service dashboards and alerts.
Feature Flag Anti-Patterns: What Breaks at Scale
Feature flags are simple in concept and accumulate technical debt faster than almost any other pattern. Flag proliferation is the first failure mode: a codebase with 500 active flags, most of which have been enabled for over a year, is harder to reason about than code with no flags. The old flag code paths still run conditionally, adding cognitive overhead and test surface area. Every old flag is a branch that must be tested in both states, doubling test permutations. The discipline of flag cleanup (removing the flag and dead code path after a successful rollout) must be explicitly scheduled, not assumed to happen naturally.
Flag debt: old flags that are permanently enabled but still in code. Schedule flag removal as a required step in every feature rollout.
Flag dependency: flag A only works correctly if flag B is also enabled. Document dependencies explicitly and test all valid combinations.
Testing without flags: every flag creates a branch. Without systematic flag-aware testing, bugs appear only in specific flag combinations.
Bootstrapping problem: the feature flag service itself must not be behind a feature flag. It must be available before any flags are evaluated.
Kill switch design: ensure every kill switch can be activated without a deployment, without waiting for any cache to expire, and by a non-engineer (operations staff) during an incident.
What Interviewers Test About Observability and Operations
Observability questions test your understanding of the difference between knowing a system is broken and knowing why. Senior engineers are expected to design systems that are observable from the start, not retrofit observability after production problems surface.
Know: the three pillars (metrics, logs, traces) and what question each answers. Metrics: is something wrong? Logs: what happened? Traces: where in the system is the problem?
Know: why distributed tracing is essential in microservice architectures (a single user request touches 10+ services; without traces you cannot see the full picture)
Know: the difference between black-box monitoring (external synthetic checks) and white-box monitoring (internal instrumentation)
Know: structured logging (JSON fields) vs unstructured logging (plain text) and why structured logging is required for log aggregation at scale
Know: how to design a feature flag rollout: start with internal users, then beta users, then 1% canary, monitor error/latency metrics, then gradual percentage ramp to 100%
OpenTelemetry: The Standard That Ended Vendor Lock-in
Before OpenTelemetry, every observability vendor (Datadog, New Relic, Jaeger, Zipkin) required you to instrument your code with their proprietary SDK. Switching vendors meant re-instrumenting every service. OpenTelemetry (OTel) emerged from the merger of OpenCensus (Google) and OpenTracing (CNCF) in 2019. It provides a vendor-neutral API and SDK: you instrument your code once with OTel, and configure which backend receives the data. Switch from Jaeger to Honeycomb by changing one config line, not by touching application code. The OTel collector runs as a sidecar or daemon, receives telemetry from services, and exports to any backend via its plugin system.
Auto-instrumentation: OTel agents automatically instrument common libraries (HTTP servers, database clients, message queues) with zero code changes.
Manual instrumentation: add custom spans with attributes for business-level context (user_id, order_id, feature_flag_value).
Propagation: OTel injects trace context (trace ID, span ID) into HTTP headers and message queue metadata automatically, connecting traces across service boundaries.
Sampling: recording every trace is expensive. Head-based sampling (sample X% of traces at ingestion) or tail-based sampling (sample all traces with errors or high latency) control cost.
Semantic conventions: OTel defines standard attribute names (db.system, http.method, net.peer.port). Consistent naming enables cross-service dashboards and alerts.
Feature Flag Anti-Patterns: What Breaks at Scale
Feature flags are simple in concept and accumulate technical debt faster than almost any other pattern. Flag proliferation is the first failure mode: a codebase with 500 active flags, most of which have been enabled for over a year, is harder to reason about than code with no flags. The old flag code paths still run conditionally, adding cognitive overhead and test surface area. Every old flag is a branch that must be tested in both states, doubling test permutations. The discipline of flag cleanup (removing the flag and dead code path after a successful rollout) must be explicitly scheduled, not assumed to happen naturally.
Flag debt: old flags that are permanently enabled but still in code. Schedule flag removal as a required step in every feature rollout.
Flag dependency: flag A only works correctly if flag B is also enabled. Document dependencies explicitly and test all valid combinations.
Testing without flags: every flag creates a branch. Without systematic flag-aware testing, bugs appear only in specific flag combinations.
Bootstrapping problem: the feature flag service itself must not be behind a feature flag. It must be available before any flags are evaluated.
Kill switch design: ensure every kill switch can be activated without a deployment, without waiting for any cache to expire, and by a non-engineer (operations staff) during an incident.
What Interviewers Test About Observability and Operations
Observability questions test your understanding of the difference between knowing a system is broken and knowing why. Senior engineers are expected to design systems that are observable from the start, not retrofit observability after production problems surface.
Know: the three pillars (metrics, logs, traces) and what question each answers. Metrics: is something wrong? Logs: what happened? Traces: where in the system is the problem?
Know: why distributed tracing is essential in microservice architectures (a single user request touches 10+ services; without traces you cannot see the full picture)
Know: the difference between black-box monitoring (external synthetic checks) and white-box monitoring (internal instrumentation)
Know: structured logging (JSON fields) vs unstructured logging (plain text) and why structured logging is required for log aggregation at scale
Know: how to design a feature flag rollout: start with internal users, then beta users, then 1% canary, monitor error/latency metrics, then gradual percentage ramp to 100%
Related reading
OpenTelemetry: The Standard That Ended Vendor Lock-in
Before OpenTelemetry, every observability vendor (Datadog, New Relic, Jaeger, Zipkin) required you to instrument your code with their proprietary SDK. Switching vendors meant re-instrumenting every service. OpenTelemetry (OTel) emerged from the merger of OpenCensus (Google) and OpenTracing (CNCF) in 2019. It provides a vendor-neutral API and SDK: you instrument your code once with OTel, and configure which backend receives the data. Switch from Jaeger to Honeycomb by changing one config line, not by touching application code. The OTel collector runs as a sidecar or daemon, receives telemetry from services, and exports to any backend via its plugin system.
- Auto-instrumentation: OTel agents automatically instrument common libraries (HTTP servers, database clients, message queues) with zero code changes.
- Manual instrumentation: add custom spans with attributes for business-level context (user_id, order_id, feature_flag_value).
- Propagation: OTel injects trace context (trace ID, span ID) into HTTP headers and message queue metadata automatically, connecting traces across service boundaries.
- Sampling: recording every trace is expensive. Head-based sampling (sample X% of traces at ingestion) or tail-based sampling (sample all traces with errors or high latency) control cost.
- Semantic conventions: OTel defines standard attribute names (db.system, http.method, net.peer.port). Consistent naming enables cross-service dashboards and alerts.
Feature Flag Anti-Patterns: What Breaks at Scale
Feature flags are simple in concept and accumulate technical debt faster than almost any other pattern. Flag proliferation is the first failure mode: a codebase with 500 active flags, most of which have been enabled for over a year, is harder to reason about than code with no flags. The old flag code paths still run conditionally, adding cognitive overhead and test surface area. Every old flag is a branch that must be tested in both states, doubling test permutations. The discipline of flag cleanup (removing the flag and dead code path after a successful rollout) must be explicitly scheduled, not assumed to happen naturally.
- Flag debt: old flags that are permanently enabled but still in code. Schedule flag removal as a required step in every feature rollout.
- Flag dependency: flag A only works correctly if flag B is also enabled. Document dependencies explicitly and test all valid combinations.
- Testing without flags: every flag creates a branch. Without systematic flag-aware testing, bugs appear only in specific flag combinations.
- Bootstrapping problem: the feature flag service itself must not be behind a feature flag. It must be available before any flags are evaluated.
- Kill switch design: ensure every kill switch can be activated without a deployment, without waiting for any cache to expire, and by a non-engineer (operations staff) during an incident.
What Interviewers Test About Observability and Operations
Observability questions test your understanding of the difference between knowing a system is broken and knowing why. Senior engineers are expected to design systems that are observable from the start, not retrofit observability after production problems surface.
- Know: the three pillars (metrics, logs, traces) and what question each answers. Metrics: is something wrong? Logs: what happened? Traces: where in the system is the problem?
- Know: why distributed tracing is essential in microservice architectures (a single user request touches 10+ services; without traces you cannot see the full picture)
- Know: the difference between black-box monitoring (external synthetic checks) and white-box monitoring (internal instrumentation)
- Know: structured logging (JSON fields) vs unstructured logging (plain text) and why structured logging is required for log aggregation at scale
- Know: how to design a feature flag rollout: start with internal users, then beta users, then 1% canary, monitor error/latency metrics, then gradual percentage ramp to 100%
