gajdos lászló

When Gajdos László's team lost $18,000 per minute during a cascading failure, he didn't just restore service-he reconstructed the entire observability pipeline to turn unknown unknowns into actionable alerts.

In the chaos of a production outage that rippled through a payment orchestration platform serving over 300,000 merchants, one SRE lead made a decision that would later influence how hundreds of engineering teams think about resilience. That lead was Gajdos László, a name that has become synonymous with pragmatic, no-nonsense observability engineering inside DevOps circles. Instead of treating the incident as a one-off, he tore apart the existing monitoring stack, pinpointed exactly where the signal-to-noise ratio had collapsed and rebuilt the entire telemetry pipeline around event-driven Service level objectives (SLOs) and strict cost-aware sampling.

This article isn't a biography; it's a deep technical analysis of the patterns, architectures, and cultural shifts that Gajdos László championed. We'll walk through the concrete tooling choices-OpenTelemetry, Tempo, Falco, LitmusChaos-and the reasoning behind them, using real-world failure modes to illustrate how his approach turns observability from a passive dashboards‑and‑pagers setup into a proactive, code‑driven safety net. If you're a senior engineer or SRE dealing with microservice sprawl, alert fatigue. Or the sinking feeling that your metrics lie to you, the lessons here are directly applicable.

The Incident That Redefined Observability at Scale

On a Tuesday at 2:14 a - and mCET, the primary Kafka cluster backing the payment processor's idempotency service split‑brained. By the time the on‑call engineer noticed-seventeen minutes later-the dead‑letter queue had accumulated 1. 2 million undelivered messages, and the business was hemorrhaging revenue. The monitoring dashboards lit up green because health checks against the /healthz endpoint passed; the Kafka Connect sink had moved to a degraded state that the existing blackbox probes didn't detect. Gajdos László described it later as "the perfect failure of checks that checked only themselves. "

The postmortem revealed a deeper problem: the team had optimized for mean time to acknowledge (MTTA) without tying alerts to actual user‑impacting conditions. They were measuring whether a pod was alive, not whether a business transaction could complete. Gajdos László immediately deprecated the legacy Prometheus alert rules that relied on static thresholds and introduced a multi‑layer validation: white‑box health endpoints that executed a lightweight synthetic transaction through the critical path, combined with request‑rate‑corrected error budget burn alerts generated from a custom OpenMetrics‑compatible sidecar. He documented the new threshold derivation in an internal guide that later became the basis for a widely‑referenced observability primer, emphasizing that every alert must answer "Does a paying customer notice? " before it can page someone,

Who Is Gajdos László and Why His Method Gained Traction

Gajdos László isn't a charismatic keynote speaker or a vendor evangelist? He's a staff software engineer who's spent 15 years in the trenches, first in Budapest's embedded Linux scene and later scaling fintech platforms in Berlin and Singapore. His methodology gained traction because he solved a visceral pain point: after every incident, teams would add more dashboards. But the same classes of failure kept escaping detection. He attacked the problem by insisting that observability signals-traces, metrics, logs. And profiles-be treated as product features, not infrastructure afterthoughts.

What sets him apart is the combination of rigorous data engineering and developer empathy. He famously coined the phrase "log cards over log lines" to advocate for structured, typed events instead of ad‑hoc printf debugging remnants. In internal projects, he enforced that every microservice emit a trace card event with a schema version, correlation ID. And a machine‑readable error code triage map. This approach influenced the OpenTelemetry semantic conventions around trace context propagation and made it possible to auto‑generate runbooks from the trace data itself. Today, his name frequently appears in SRE slack channels when someone shares a dashboard that "actually shows you why the checkout flow broke. "

Observability as Code: Rebuilding the Telemetry Stack with OpenTelemetry

After the Kafka incident, Gajdos László pushed for a complete rewrite of the telemetry layer using OpenTelemetry (OTel) as the single collection and export API. The previous stack had three separate agents-Jaeger for traces, Prometheus Node Exporter for metrics. And Fluent Bit for logs-each with its own configuration drift. By moving to the OTel Collector with a tree of processors, he unified sampling, tail‑based trace analytics and metric aggregation into a single pipeline defined in YAML, stored in Git alongside the application code.

The real innovation was in the sampling strategy. Instead of the simple head‑based 1% sampling that misses rare errors, Gajdos László implemented a two‑tier system: a head‑based probability that ensures a statistically significant sample of all traffic. And a tail‑based policy that catches any span whose status code is ERROR or whose duration exceeds 99th percentile latency plus three standard deviations. The OTel Collector's tailsamplingprocessor was configured with a composite policy that also forces sampling on spans containing the custom attribute payment idempotency risk=high, because that attribute was known to precede dead‑letter growth. The result was a 15x reduction in trace storage cost while preserving every pathological transaction for debugging. For any engineer considering a similar migration, pay attention to the collector's memory‑limiter and batch processor settings; the default values will cause OOM kills under 10k spans/sec.

The Chaos Engineering Gambit: Validating Alerts Before They're Needed

Finding broken alerts during an incident is too late. Gajdos László embedded chaos engineering directly into the CI/CD pipeline, making every alert rule pass a litmus test before deployment. He used LitmusChaos to inject faults-network latency, pod deletions, Kafka partition leader switches-in staging clusters that mirrored production topology, then verified that the intended alert fired and that no unexpected secondary alerts exploded the pager.

One notable experiment simulated a graduated payment timeout: 200ms latency injected for 5% of requests to the acquirer simulator, stepping up to 500ms. The SLO‑based alert for payment latency had a one‑minute burn rate window; the test confirmed it fired at the correct multi‑burn rate threshold and that the runbook automatically executed the circuit breaker that shifted traffic to a secondary provider. This "alert‑driven resilience playbook" philosophy is now codified in his team's LitmusChaos experiments repository. The key lesson: if you can't trigger your alert on purpose in a controlled environment, assume it's false confidence. Gajdos László recommends scheduling these gamedays weekly; his teams discovered that 22% of alerting rules in the original migration were either incorrectly thresholded or completely silent.

How Gajdos László Transformed Incident Response with Event-Driven SLOs

Traditional SLO dashboards are laggy averages; you might breach your error budget for an hour before a scheduled report triggers a ticket. Gajdos László moved SLO evaluation into the stream processor, using Kafka Streams to compute burn rate in real time over sliding windows and conditionally forward alert events to PagerDuty only when a burn rate multiplier exceeded 14. 4x (meaning you'd consume the budget in 1/14. 4 of the window, a standard multi‑window approach).

He built a custom operator called "SLO‑gate" that consumed the same metrics stream and, when a critical burn rate was detected, dynamically turned on 100% error tracing for the affected service and increased log level to TRACE for the next five minutes. This eliminated the painful cycle of "the issue is gone by the time you look. " Engineers investigating the alert could open the trace visualizer and see the exact cascade with full data. Because the system had already reacted. The architecture is documented in detail in his internal tech report, but the core idea is reproducible: decouple metric evaluation from static instrumentation, and let the SLO engine modulate observability intensity based on signal health. This pattern reduces mean time to resolve (MTTR) for many teams by over 40%, according to Gajdos László's post‑deployment surveys.

Deploying Distributed Tracing across Polyglot Microservices: Lessons from the Trenches

Any engineer who has tried to propagate trace context through a Ruby‑on‑Rails monolith, a Node js frontend, and a Rust performance‑critical service knows it's a mess. Gajdos László's team faced exactly this: the payment gateway used Spring Boot, the risk scoring engine ran on Actix Web, and the merchant dashboard was Next js. He enforced W3C Trace‑Context headers at the API gateway and wrote a library‑agnostic integration test suite that verified every service's outgoing headers and span relationship integrity using the OTel Java Testbed and equivalent Rust instrumentation.

The biggest hurdle was asynchronous work-Kafka consumers that process messages without an incoming HTTP request. Gajdos László solved this by adding a custom Kafka header propagator that injected the trace context from the producer's record header, allowing downstream spans to link correctly. He also mandated that every consumer set a span kind of CONSUMER and a link to the producer span, so the tracing UI would render a clear dependency chain. For mobile frontend monitoring, he paired the OpenTelemetry JS SDK with a custom sampling rule that forces tracing for any user session that generates an HTTP 5xx, sending the trace parent to a backend collector via the native app's network interceptor. (For mobile‑first architectures, see our mobile app monitoring integration guide for patterns that mirror this approach. ) The hard‑won insight: test your propagation with nested async workers, not just simple request/response pairs.

From Blackbox to Glassbox: Instrumenting Legacy Monoliths Without Rearchitecting

Not every system can be re‑platformed. The payment orchestration platform still relied on a 15‑year‑old Java monolith for settlement files. Gajdos László used OpenTelemetry Java agent auto‑instrumentation to capture spans from the JDBC calls and thread pools without touching the source code. But the real breakthrough was adding a custom span processor that intercepted the internal state machine transitions and emitted structured events with domain‑specific attributes like settlement batch id and reconciliation, and discrepancyamount. While

By treating the legacy system as a glassbox, the team could finally correlate batch failures to upstream payment declines. They discovered that 34% of settlement delays stemmed from a single thread starvation issue when the reconciliation service called an external FTP that timed out. A simple change to a fixed thread pool size and a circuit breaker reduced settlement failures by 91%. The lesson: instrumentation without refactoring is valid and often the fastest path to reliability. Gajdos László's rule of thumb is that any service older than three years that processes money must have at least a "telemetry catheter" inserted-OpenTelemetry agent plus a sidecar that exports JVM metrics-before any team is allowed to claim "we don't