incydent

In the aftermath of a 4 a m. Sev-0 outage, the first question isn't "what broke" - it's "why did our response pipeline take 12 minutes to wake up the right person? " Most Incident management tooling still treats on‑call schedules, alert routing, and remediation steps as isolated configuration silos. Incydent replaces that fragmented approach with a unified, declarative model where incident pipelines are version‑controlled artifacts - just like your Terraform plans or Kubernetes manifests. Over the past six months, evaluating Incydent across three production clusters handling 800,000 requests per minute forced our team to rethink how incident response should intersect with observability, runbook automation, and SRE culture. This article digs into the architecture, real‑world tradeoffs. And the engineering decisions behind Incydent's event‑driven core. While showing how it stacks up against entrenched incumbents like PagerDuty and Opsgenie.

Why Traditional Incident Management Falls Short

Most on‑call rotation tools were born in the pre‑Kubernetes era. PagerDuty launched in 2009, Opsgenie in 2012 - long before Prometheus scrape intervals hit 15 seconds and service meshes turned every pod into a potential failure domain. The data model of these platforms treats incidents as tickets with a lifecycle but the surrounding machinery - alert deduplication - context enrichment, diagnostic command execution - lives in external scripts, webhook glue. And Slack bot forests. The result is what we've come to call "config sprawl death": by the time a third‑party service experiences latency, twenty different micro‑services have fired alerts. And no single system can correlate them without human triage.

In production environments, we found that 40% of high‑urgency pages were either false positives or duplicates. Each noisy alert diluted the team's trust in the paging system, leading to longer mean time to acknowledge (MTTA). The root cause is architectural: traditional incident managers are passive receivers, not active participants in the observability stack. They lack native support for OpenTelemetry context propagation or structured runbook execution. Incydent was designed to address this gap by treating an incident as a state machine triggered by observable signals, not just a human‑opened ticket.

Introducing Incydent: A Declarative Approach

Incydent is an open‑core incident orchestration platform that defines response logic as code. Rather than clicking through a UI to define escalation policies, you write YAML‑based incident pipelines that describe trigger conditions, enrichment sources - routing rules. And automated remediation steps. These pipelines are stored in a Git repository alongside the service code, which means they get reviewed, tested, and deployed through CI/CD - a practice our site reliability engineers have been advocating since the publication of the Google SRE book's chapter on managing incidents.

The mental model shift is significant. Incydent isn't a ticketing system with integrations; it's an engine that consumes metrics, logs, and traces, applies a configurable ruleset, and orchestrates a response. For example, a pipeline might specify: when the error budget burn rate exceeds 5% over 10 minutes for the checkout service, create an incident, attach a PagerDuty alert (if still using that as a transport), run a diagnostic playbook against the Kubernetes cluster. And post a structured summary to Slack. All of that's expressed in a single document, versioned, with drift detection if someone manually edits the in‑memory state. This declarative pattern has been battle‑tested in infrastructure‑as‑code tools like Terraform; Incydent brings it to incident lifecycle management.

Architectural Deep‑Dive: Event‑Driven Pipelines

Under the hood, Incydent runs a lightweight, horizontally scalable engine written in Go, with a persistence layer designed around event sourcing. Each Incident object is an append‑only log of domain events - AlertFired, Acknowledged, Diagnosed, Escalated, Resolved - which means you can replay the entire signal chain for post‑mortem analysis. This event‑sourced design borrows from patterns described in Martin Kleppmann's "Designing Data‑Intensive Applications" and is particularly powerful when debugging why a given alert didn't escalate properly.

The core pipeline is driven by a CEL (Common Expression Language) based rule evaluator, the same expression language used in Kubernetes Validating Admission Policies. This allows SREs to write complex conditions without embedding a full scripting runtime. For instance, a condition can check if `rate(http_requests_total{status=~"5. "}5m) > 0, and 1 AND oncallschedule("primary"). time_between("02:00","06:00")`, while the integration layer, built on a plugin‑oriented architecture, connects to Prometheus, Alertmanager, Grafana, Datadog, and custom webhooks. Each notification is normalized into an Incydent alert event, stamped with a unique fingerprint to handle deduplication - a critical feature often bolted on after the fact in PagerDuty.

Integrating Incydent with Prometheus and Alertmanager

In our stack, we already ran Alertmanager as the central alert router. Instead of replacing it, Incydent sits between Alertmanager and human responders, acting as an intelligent layer that can suppress, enrich, or auto‑remediate before waking anyone. The Incydent‑Alertmanager bridge uses the Alertmanager webhook receiver and the v2 API, consuming Firing and Resolved events. What sets Incydent apart is its ability to pull additional metrics from Prometheus at incident open time - for example, fetching the node memory pressure metric to diagnose whether a pod eviction is imminent.

By interpreting Prometheus recording rules and alert labels, Incydent builds a dynamic topology of affected services. This enables correlation that would otherwise require a separate tool like Cortex or Thanos. During a recent database connection pool exhaustion incident, Incydent identified that three front‑end services sharing the same connection pool label `app=orders` were firing simultaneously, grouped them into a single incident. And ran a predefined playbook that scaled up the reader replicas. The mean time to resolution (MTTR) dropped from 23 minutes to 7, simply because the right context was surfaced before the first human acknowledged the page. This kind of tight integration requires careful Prometheus metric naming conventions; our team documented them in our observability standards guide to keep things consistent.

The Role of OpenTelemetry Context Propagation

One area where Incydent truly differentiates itself is its native support for distributed tracing context. When an alert originates from a span that has a trace ID and span ID - typical in microservices instrumented with OpenTelemetry - Incydent can attach the entire trace waterfall to the incident timeline. This capability closes the loop between proactive observability and reactive incident response, which most vendors treat as two separate universes.

During a performance degradation in a payment processing service, an Incydent trigger fired from a Datadog anomaly monitor. Because the service emitted W3C trace context headers, Incydent automatically linked the incident to a specific trace that showed a 3‑second delay in a `POST /authorize` call due to a slow downstream fraud check. The on‑call engineer didn't have to jump between Jaeger and PagerDuty; the incident console embedded the trace flame graph. This integration leans heavily on the W3C Trace Context specification and requires that the OTel collector export spans to a backend that Incydent can query. Our implementation uses Tempo with a JSON API.

On‑Call Automation and Escalation Policies as Code

Escalation policies in Incydent are not static lists of email addresses; they're programmable resources. Using a syntax reminiscent of Kubernetes operators, you can define time‑based escalation, stakeholder notification chains. And even automated pausing during maintenance windows. For example, an escalation policy might specify: "Notify the primary on‑call for team‑checkout; if no acknowledgment in 5 minutes, page the secondary and post a critical message to the #incidents‑warroom channel; if still unresolved after 15 minutes, invoke the `rollback-last-deploy` pipeline. "

Our team moved away from managing PagerDuty escalation policies in a web UI after an undocumented change caused alerts to go into a black hole during a weekend deploy. With Incydent, all policies are stored in a GitOps repository and applied via a reconciliation loop, similar to Flux or ArgoCD. This means any change to an on‑call rotation or escalation rule gets a pull request, a peer review. And a dry‑run validation against the production Incydent server. The audit trail is immutable. Which satisfies compliance requirements under SOC 2 and ISO 27001 without additional tooling. For larger organizations, this policy‑as‑code approach also enables templating: a base escalation policy can be extended per team, dramatically reducing the overhead of managing 200 microservices.

Comparing Incydent to PagerDuty and Opsgenie

From a feature‑by‑feature standpoint, PagerDuty still leads in ecosystem breadth - over 700 integrations - and its mature mobile app for immediate acknowledgment. Opsgenie, now part of Atlassian, offers excellent Jira integration and a powerful global alert routing model. However, both tools are fundamentally closed‑source SaaS products where the incident logic is opaque. Incydent's open‑core model allows teams to self‑host the engine on their own Kubernetes clusters. Which is a non‑negotiable requirement for industries with strict data residency laws or highly regulated environments like fintech and healthcare.

For performance, we ran Incydent in a 3‑node cluster behind a load balancer, handling 2,000 alert events per minute with sub‑second processing latency. PagerDuty's event API can handle similar throughput. But we've encountered throttling during large‑scale outages - a risk when thousands of Prometheus alerts fire simultaneously. Incydent's ability to deduplicate and suppress at the edge, via its built‑in grouping window, reduces the blast radius before an incident is even created. The tradeoff is operational overhead: you're now responsible for running Incydent's dependencies (Redis, PostgreSQL, and an event store) which, if not managed well, could become a single point of failure. That's why we recommend pairing it with a reliable Kubernetes storage backup strategy.

Security and Compliance in Incident Response

Incidents often involve sensitive data - error logs might contain PII. And incident timelines could reveal system vulnerabilities. Incydent's security model is built around zero‑trust principles: all API access requires mTLS or OIDC tokens. And data at rest in the event store is encrypted with keys managed by a KMS plugin. The pipeline definitions support secret references via HashiCorp Vault or Kubernetes Secrets, so runbook commands never hardcode credentials.

Compliance automation is another strong suit. Because every incident action is an event in an append‑only store, generating audit reports for regulators becomes a query rather than a forensic exercise. The system can export a signed, chain