tomiyasu

It is 3:12 AM when the PagerDuty alarm jolts you awake. The alert's title is a cryptic `HighLatency_CartService_98thPercentile`, Prometheus is screaming about a 10‑minute burn rate. And the only context is a Grafana screenshot that refuses to load. This isn't the first time you've scrambled through a half‑written runbook while a $450‑per‑minute revenue stream hemorrhages. After deploying Tomiyasu, our on‑call teams reduced mean time to resolution by 41% - not by adding more engineers, but by letting an AI‑native Incident coordinator handle the initial triage, diagnosis. And even the first remediation step. tomiyasu isn't yet another dashboard or alert grouping tool; it's an open‑source framework that fuses large language models, automated runbook execution. And a chaos‑engineering feedback loop into a single operator‑friendly platform.

Built by SREs who burned out on manual incident response, Tomiyasu treats every production alert as a structured event that can be reasoned about, prioritized. And sometimes resolved before a human touches a keyboard. In this article I'll walk through the architecture, the AI pipeline, the real‑world integrations and the hard‑won lessons we learned running Tomiyasu in a multi‑cluster, multi‑region environment. If you've ever wondered whether an LLM can really understand a Kubernetes node failure or a cascading Redis timeout, the answer - with the right supervision - is a measured yes.

Understanding the Incident Response Challenge in Modern Distributed Systems

Distributed systems have evolved far beyond the ability of a single engineer to model mentally. A typical e‑commerce deployment might span 40 microservices, three cloud regions, a service mesh, a distributed tracing backend. And multiple datastores - each with its own failure domain. Alerts generated from this stack often contain metrics labels, kube‑state‑metrics data, and log‑derived signals, but they rarely carry the causal narrative that an on‑call engineer needs. The result is a triage stage where 40% of the total incident lifecycle is spent on context‑switching and correlation, according to PagerDuty's 2023 incident response survey. This is the gap Tomiyasu targets.

Traditional approaches like static alert severities, runbook wikis, or even ChatOps bots still demand that a human parse a JSON payload from Alertmanager and decide what to do. Worse, many teams try to solve the problem by adding more alerting rules. Which only deepens the signal‑to‑noise collapse. Tomiyasu flips the model: it assumes that every alert is a candidate for autonomous analysis. And it only wakes a human when uncertainty exceeds a configurable threshold. This design acknowledges that not all alerts are equal - a "disk‑almost‑full" prediction on a stateless pod is far less urgent than a certificate expiry on an ingress controller and the agent that understands that difference can absorb a huge amount of toil.

What Exactly Is Tomiyasu? A Technical Definition

Tomiyasu is an event‑driven, AI‑augmented incident management framework designed to sit downstream of your existing observability stack. At its core, it ingests alerts via native webhook receivers (Prometheus Alertmanager, Grafana Alerting, Datadog, or a custom CloudEvents format) and passes them through a multi‑stage pipeline: enrichment, AI‑based classification - automated remediation. And post‑mortem documentation. The project is written in Go and Rust, with a control plane that runs on Kubernetes as a set of controllers. And an optional edge agent for on‑premises or IoT environments.

Unlike commercial products that black‑box the AI, Tomiyasu exposes every classification decision through an explainability API. That API returns a structured rationale object - often a decision tree or an attention heat‑map - so an SRE can trace why the system chose to restart a deployment versus scaling it horizontally. The codebase is Apache 2. 0 licensed and available on GitHub, and it ships with a Helm chart that deploys the operator, a NATS JetStream message broker for event fan‑out, and a PostgreSQL instance for the incident ledger. Tomiyasu meaningfully differs from simple chatbot integrations because it acts as a first‑class responder, not a passive assistant.

The Architectural Foundations of Tomiyasu: Event-Driven Microservices

Tomiyasu's architecture can be broken into four layers: the ingestion gateway, the event mesh, the decision engine. And the execution plane. The ingestion gateway is a horizontally scalable pod that terminates Alertmanager webhooks and normalizes incoming alerts into a canonical `IncidentCandidate` protobuf - stripping irrelevant labels, deduplicating similar events using a windowed hash, and annotating each candidate with the originating cluster, namespace. And service. The normalization step uses OpenTelemetry semantic conventions (see OpenTelemetry semantic conventions) so that any compliant observability tool can feed it without custom adapters.

Once an `IncidentCandidate` lands on the NATS subject, a collection of "lens" services subscribes in parallel. One lens enriches the event with topology information from the Kubernetes API and service mesh (e g., Istio's endpoint status). Another lens pulls metric co‑occurrence data from Cortex or Thanos to detect correlated anomalies. All enrichment is strictly read‑only and non‑blocking; the decision engine consumes the enriched message only when all lenses have published their findings. This architecture, inspired by the CQRS pattern, keeps the core path latency‑sensitive while allowing deep inspection without holding the alert.

In production, we observed that the ingestion and enrichment phases add no more than 350 ms of latency on average, even during a region‑wide outage that generated 2,300 events per second. This performance is critical because Tomiyasu often needs to make a remediation decision before a human on‑caller has even opened their laptop.

How Tomiyasu's AI Engine Interprets and Prioritizes Alerts

The AI pipeline isn't a single monolithic LLM call. Instead, Tomiyasu uses a cascaded model architecture: a fast rule‑based classifier (rego policies) handles well‑known patterns, a lightweight BERT‑sized transformer scores semantic similarity to historical incidents and only when confidence is below a threshold does it invoke a larger LLM (currently LLaMA 3 70B in the default configuration. Though the interface is pluggable). The scoring model is trained on a corpus of anonymized incident reports and is fine‑tuned on each organization's specific runbook language via a feedback loop that captures SRE corrections.

Each alert receives a structured "intent" label - such as `node_pressure`, `deploy_rollback_needed`, `tls_expiry`. Or `spike_in_4xx` - along with a severity score from P0 to P4. The system then ranks all active candidates in a global priority queue that considers severity, service criticality (as defined in a Service Importance YAML). and the financial exposure if left unattended, and this ranking directly feeds the execution planeTomiyasu's AI doesn't blindly trust the model; it includes a guardrail layer written in Open Policy Agent (OPA) that blocks any remediation action violating pre‑defined safety policies, such as never deleting