sloop

When an incident in a Kubernetes cluster ends, the hardest question is rarely "what changed? " it's usually "what changed, in what order, across which controllers,? And who was looking at the wrong time? " Traditional logs tell you what a process emitted. They don't reconstruct the State of a Deployment - a ConfigMap. Or a Service as it shifted between two broken states.

That reconstruction gap is exactly what Sloop targets. Sloop is an open-source Kubernetes History visualization tool that records resource changes as they flow through the API server and presents them on a searchable timeline. I have used it in production troubleshooting sessions where kubectl get events returned nothing useful and audit logs were too noisy to parse quickly.

The real value of Sloop isn't just seeing that a resource changed - it's being able to replay how a cluster arrived at a broken state without hand-assembling scattered API objects. This article is a senior engineer's deep explore what Sloop does, where it fits, and where it falls short.

What the Sloop Project Actually Does

Sloop watches Kubernetes resources through the API server's watch interface, records every observed state transition. And exposes those transitions through a web-based timeline. Rather than treating Kubernetes events as a stream of disconnected log lines, Sloop treats each object as a series of versions. When you select a Deployment, you see exactly when the pod template changed, what the previous replica count was. And which fields were modified between revisions.

The project was originally developed and open-sourced by Salesforce engineering to solve a recurring problem: Kubernetes has excellent primitives for desired state but no built-in way to visually inspect how actual state has moved over time. The backend is written in Go and the frontend is a lightweight React application it's intentionally local-first - Sloop runs inside the cluster, consumes the watch stream. And persists compressed snapshots to local disk.

This design means Sloop doesn't depend on an external database or a separate analytics service. You install it, point it at the API server. And start accumulating object history immediately. For a single namespace or a small cluster, the operational footprint is small enough that it can run as a sidecar-style debug tool without disturbing production workloads.

The Observability Gap That Sloop Addresses

Standard Kubernetes observability stacks are good at metrics and pretty good at logs. But they're weak at object-level change forensics. Prometheus can tell you that a Deployment's pod count dropped to zero. It can't easily tell you that a ConfigMap key was deleted by a broken admission webhook, then restored by an operator, only to be overwritten again by a CI job. That sequence matters when you're troubleshooting a cascading failure.

The Kubernetes event mechanism isn't a history store. In most clusters, the API server retains events for a short default window - often one hour - and events contain limited structured data about the object they reference. A Warning event like FailedScheduling tells you that a Pod couldn't be scheduled. But it doesn't give you the full Pod spec or show what changed in the node selector ten minutes earlier.

Sloop fills that gap by retaining object state over a much longer horizon and organizing it by resource identity that's closer to a time-series database for Kubernetes objects than to a log aggregator. If your current setup relies on Kubernetes observability stack planning to choose tools, Sloop belongs on the shortlist for change-history tooling.

Installing Sloop in a Production Kubernetes Cluster

Installation starts with the official manifests in the Sloop source repository. The deployment creates a single Pod that runs the Sloop server plus the watcher. It needs a ServiceAccount with read access to the resource types you want to track. In most cases, you can start with a ClusterRole that grants get, list, watch on Deployments, Pods, Services, ConfigMaps. And ReplicaSets.

Before enabling it in production, consider the storage path. Sloop writes its local store to an emptyDir or a PersistentVolume. If you use emptyDir, the history disappears when the Pod restarts. In production, use a small PersistentVolumeClaim and set a retention policy that matches your incident review window. A 10 GiB volume is usually enough for several weeks of change history in a moderately active namespace. But the right number depends on object churn.

Once the server is running, you can expose it through a port forward or a restricted Ingress don't expose the Sloop UI to the public internet without authentication. The tool doesn't implement built-in identity and access management. So place it behind your existing SSO proxy or Kubernetes RBAC-aware ingress controller. For a secure internal view, kubectl port-forward is the simplest starting point.

Reconstructing Incident Timelines with Sloop's Web Interface

The web interface is built around a timeline panel on the left and a detail pane on the right. You filter by namespace - resource kind, and resource name. The timeline shows vertical markers for each observed change. Clicking a marker loads the full object as it existed at that revision - including annotations, labels. And nested pod templates.

In a recent production incident, I needed to find out why a NodePort Service suddenly stopped routing traffic to healthy pods. The Service object hadn't been manually edited according to our GitOps history. But the Sloop timeline showed a three-minute window where a controller changed the selector from app: checkout to app: checkout-canary. That change was invisible in the Git repo and in the standard event stream. But it was obvious once we saw the selector diff in the detail pane.

The interface also supports comparing any two revisions. Sloop renders a field-level diff using merge patch semantics. Which avoids the noise of a raw YAML dump. If you have ever manually compared two exported Kubernetes YAML files during an outage, that diff view alone is worth the installation effort.

How Sloop Stores Resource State Differently

Sloop's storage model is closer to a local event-sourced database than to a traditional log file. Each watch event is parsed into a structured object and appended to an embedded key-value store. The key includes the resource identity, the resource version, and a timestamp. The value contains the serialized object plus metadata about the watch event type.

This append-friendly design matters because Kubernetes watches aren't guaranteed to deliver every intermediate state if connectivity drops. The API server may send a BOOKMARK event or a ERROR event. And the client must re-list and reconcile. Sloop handles those watch semantics by tracking the resource version and re-syncing when needed, as described in the Kubernetes API watch documentation.

Unlike etcd. Which compacts history to preserve performance, Sloop's local store can retain object revisions for as long as you have disk space. That gives you a longer forensic window without changing the API server's storage configuration. However, it also means you're now responsible for capacity planning, backup, and retention - operational concerns that etcd normally handles for you.

Comparing Sloop to kubectl, Events, and Audit Logs

The quickest way to understand Sloop is to compare it to the tools most engineers already use during an incident:

• kubectl get events: shows recent event objects. But events are short-lived and don't include full object state.
• Kubernetes audit logs: capture API requests and responses with user attribution, but require an external backend and generate high volume.
• Sloop: focuses on object state over time, with visual diffs and a searchable web UI. But doesn't record the authenticated user who made each request.

Audit logs remain the authoritative source for "who did what" from a security perspective. Sloop answers a different question: "what did the resource look like before and after? " In a compliance investigation, you often need both. Sloop can help you locate the exact moment a change occurred. And then you can pull the corresponding audit entry by timestamp and resource version.

If you're already piping audit events to an external system, consider Sloop as a complementary layer rather than a replacement. The Kubernetes audit logging documentation explains how to configure audit levels and backends. Sloop is much easier to query for object diffs than raw audit JSON.

Real-World Forensics: A Node Autoscaler Failure

In one production cluster, a HorizontalPodAutoscaler kept reporting that a Deployment was already at its maximum replica count, yet the Deployment only had two pods running. The HPA status showed a target of eight, the Deployment spec showed six replicas. And the cluster autoscaler refused to add nodes because it believed the pending pods weren't schedulable. Every individual object looked correct when viewed in isolation,

Sloop revealed the sequenceAt 09:42, a config management operator updated the Deployment's resource requests to request more CPU. At 09:44, the HPA controller updated the Deployment's replica count based on the old resource model. At 09:46, another operator rolled back the resource request change. But left the replica count at the inflated value. The cluster autoscaler saw inconsistent capacity signals and stopped scaling nodes.

Without Sloop, we would have spent hours exporting YAML from Git, querying the API server. And guessing which controller changed what. With the timeline open, the root cause was visible in about ten minutes that's the practical difference between object history and log history,

Operational Risks and Limits of Historical Visualization

Sloop isn't a silver bullet. Because it records object state, it can inadvertently capture sensitive data. If your RBAC grants Sloop read access to Secrets, the tool will store secret values in its local database. You should either exclude Secrets from the watched resources or treat the Sloop data directory as highly sensitive. In production, I explicitly deny Secret access to the Sloop ServiceAccount,

The single-replica architecture is another limitationIf the Sloop Pod restarts and its storage volume is lost, the history disappears. And there's no built-in high-availability mode or replicationFor teams that need durable change history across node failures, Sloop should be backed by a PersistentVolume and included in regular volume snapshots.

Sloop also doesn't attribute changes to human users or external systems by itself. Kubernetes objects contain managedFields. But those often point to controller names like kube-controller-manager rather than a responsible engineer. If you need strict change attribution, pair Sloop with audit log correlation or a GitOps audit trail.

Extending Sloop for Compliance and Change Tracking

Compliance frameworks often ask you to prove that a change was reviewed, approved, and implemented as expected. Sloop can provide the implementation side of that evidence by showing exactly what changed on the cluster and when. For example, if a PCI requirement says you must monitor changes to network policy objects, Sloop gives you a dated, diffable record.

To make that evidence admissible, you need two additions. First, restrict write access to the Sloop UI and data directory so the history cannot be tampered with. Second, export or snapshot important revisions to an immutable object store. A simple cron job can use the Sloop API to pull revisions and write them to S3 with object lock enabled.

Some teams extend Sloop by feeding its output into CI checks. For example, you can write a script that compares the current Deployment against the last approved revision in Sloop. If the diff contains unexpected fields, the pipeline fails. That closes a gap between desired-state enforcement and actual-state history that GitOps alone doesn't always catch.

Sloop in a GitOps and Continuous Delivery Pipeline

GitOps assumes that Git is the source of truth. But the cluster is where truth actually happens. Sloop gives you a way to verify that the cluster followed the intended path. When a sync runs, Argo CD or Flux updates resources. Sloop records those updates as distinct revisions. You can correlate a Git commit SHA with a Sloop revision marker to prove that a rollout reached the cluster.

In continuous delivery, a failed rollout often leaves behind a resource that was partially updated. Sloop's diff view is faster than comparing Helm values files or kustomize overlays. You can jump directly to the revision where the Deployment's image changed, then compare the new replica set's pod template with the previous one that's especially useful when a canary deployment interacts badly with an HPA or a PodDisruptionBudget.

If you operate a platform that manages multiple tenant namespaces, Sloop can help you answer the inevitable "what happened to my namespace? " question. Point tenants to a read-only Sloop view scoped to their namespace. And you reduce the number of direct API server queries while giving them a visual timeline. Just be careful to scope RBAC and data retention per tenant.

Frequently Asked Questions About Sloop

Is Sloop a replacement for Kubernetes audit logging,

NoSloop records object state changes for visual forensic analysis, but it doesn't capture authentication details, request metadata. Or all API server interactions. Audit logs remain necessary for security investigations and compliance. Use Sloop alongside audit logs, not instead of them.

Does Sloop store Secrets,? But

Only if you grant the Sloop ServiceAccount permission to read Secrets and add Secrets to the watched resource list? In production, you should exclude Secrets or explicitly deny access. Even if you do grant access, the data is stored locally and has the same sensitivity as the Secret itself.

How long does Sloop retain resource history?

Retention is determined by the storage capacity you allocate and any pruning logic you configure. Sloop doesn't impose a hard default time limit on all versions. Many teams keep several weeks of history on a small PersistentVolume. But high-churn namespaces will need more space or shorter retention.

Can Sloop monitor multiple clusters?

Sloop is designed to run inside a single cluster and watch that cluster's API server. To monitor multiple clusters, you can deploy one Sloop instance per cluster. Or run Sloop on a management cluster and configure it to watch remote API servers if network and RBAC allow there's no built-in multi-cluster aggregation in the core project,

Is Sloop production-ready for large clusters

It depends on object churn and storage capacity. Sloop works well in small to medium clusters with moderate change rates. In very large clusters with thousands of objects changing per minute, the local store can become a bottleneck. Benchmark it in a staging environment before rolling it out broadly.

Conclusion: Sloop Turns State History Into a Debugging Asset

Sloop succeeds because it treats Kubernetes resource state as a first-class forensic artifact. Most observability tools focus on metrics, logs, or traces. Sloop focuses on the object itself - the Deployment, Service. Or ConfigMap that changed underneath a healthy process graph. That focus makes it uniquely useful when an incident is caused by configuration drift, controller conflicts. Or a broken GitOps sync.

If your team is still assembling incident timelines from scattered YAML files and incomplete event streams, install Sloop in a staging cluster and point it at a busy namespace. You will likely find the answer to a past mystery within the first hour of browsing. Then decide whether the operational overhead of retention, access control. And storage is worth the faster root-cause analysis. For most teams, it is.

Try Sloop in your environment, and if you find it useful, consider contributing to the project or documenting your setup for the community. The tool is open source and benefits from real-world forensics patterns. For broader cluster telemetry guidance, see Kubernetes observability stack planning and SRE incident forensics checklist.

What do you think?

Is object-level state history more valuable than traditional log aggregation for Kubernetes incident response,? Or does it duplicate what audit logs already provide?

Should Sloop add built-in authentication and multi-cluster support,? Or is it better to keep the tool focused on single-cluster local forensics and let external proxies handle access control?

Would you trust a local-first history store like Sloop as compliance evidence, or do immutable exports and audit log correlation remain mandatory before it can be used in regulated environments?