sunfish

When Nature Inspires Code: Unpacking the Sunfish Pattern in Distributed system

At first glance, the word "sunfish" might evoke images of a slow-moving, disk-shaped marine creature basking near the ocean surface. But for those of us working in distributed systems, observability. And cloud-native architecture, the term has taken on a very different meaning. In engineering circles, the "sunfish" pattern refers to a specific anti-pattern in service mesh and microservice communication-one where a single component becomes a monolithic bottleneck, absorbing traffic from all directions while offering little in the way of fault isolation. If your microservice architecture is starting to look like a sunfish, you might be heading for a production meltdown.

The original sunfish (the fish) is known for its large, flattened body and minimal swimming ability. It drifts with currents, often becoming a host for parasites. In software, a sunfish service does the same: it drifts through the architecture, accumulating dependencies, handling cross-cutting concerns like authentication, logging. And rate limiting. And eventually becoming the single point of failure. I've seen this pattern emerge in production environments where teams prioritize speed of delivery over architectural discipline. The result is a service that everyone depends on but no one wants to own.

This article will dissect the sunfish anti-pattern from a technical perspective. We'll look at how it manifests in Kubernetes clusters, why it's dangerous for SRE teams. And what tools like Envoy, Linkerd. And OpenTelemetry can do to mitigate it. We'll also explore real-world examples from open-source projects and RFCs that define best practices for service mesh topology. By the end, you'll have a concrete framework for identifying and refactoring sunfish services in your own stack.

Defining the Sunfish Anti-Pattern in Microservice Architecture

The sunfish anti-pattern is characterized by a single service that handles an outsized proportion of inter-service communication. In a typical service mesh, traffic flows through sidecar proxies (like Envoy) that handle routing, retries. And observability. But when a sunfish service emerges, it bypasses the mesh or becomes a centralized hub that all other services must call to perform basic operations. This creates a topology that looks like a star network, with the sunfish at the center.

From a data engineering perspective, this pattern breaks the principle of least privilege and increases blast radius. If the sunfish service goes down, every downstream service is affected. I've seen this in a production Kubernetes cluster where a "gateway" service handled authentication, authorization. And request transformation for 40+ microservices. When that service experienced a memory leak during a traffic spike, the entire platform went dark for 12 minutes. The postmortem revealed that the team had intentionally centralized logic to "simplify" the architecture,, and but they had inadvertently created a sunfish

The key identifiers of a sunfish service include: high fan-in (many services calling it), high fan-out (it calls many services), low cohesion (it handles unrelated concerns like auth, logging. And caching). And a lack of clear ownership. In the RFC 9205 on HTTP API design, the authors emphasize that APIs should be designed for evolvability and independence. A sunfish service violates this principle by coupling multiple concerns into a single deployment unit.

Why the Sunfish Pattern Is Dangerous for Observability and SRE

For SRE teams, observability is the first casualty of a sunfish architecture. When a single service handles traffic from dozens of others, it becomes impossible to attribute latency or errors to specific upstream or downstream components. In one incident I worked on, a team spent three days debugging a 5-second latency spike in their payment pipeline. The root cause was a sunfish service that had accumulated a blocking I/O call from a legacy database connection pool. Because the service handled both payment and user profile requests, the latency was visible across multiple endpoints, making it look like a systemic issue rather than a localized one.

OpenTelemetry can help. But only if the sunfish service is instrumented correctly. In practice, many teams skip distributed tracing for internal services, assuming that the mesh will handle it. But when a sunfish service is the only path between services, traces become incomplete or misleading. I recommend using the OpenTelemetry trace specification to enforce consistent span propagation across all services, including the sunfish. If the sunfish service cannot be refactored immediately, at least instrument it with context propagation to make debugging less painful.

Another risk is that sunfish services often become configuration magnets. Teams add feature flags, routing rules, and rate limits directly into the sunfish service. Because it's the easiest place to put them. This leads to a configuration surface that's both fragile and opaque. In a production environment, we found that a single sunfish service had over 200 environment variables, most of which were undocumented. A simple typo in a configmap caused a cascading failure that took down three dependent services.

Real-World Examples: Sunfish in Kubernetes and Service Meshes

The sunfish pattern is particularly common in Kubernetes environments where teams use Ingress controllers or API gateways as a catch-all. For example, a team might deploy an NGINX Ingress controller that handles TLS termination, rate limiting, authentication, and request rewriting. While this works for small clusters, as the number of services grows, the Ingress becomes a sunfish. I've seen a production cluster where the NGINX Ingress handled 85% of all east-west traffic (internal service-to-service calls) because developers configured it as a reverse proxy for every internal API.

Service meshes like Linkerd and Istio were designed to prevent this exact scenario. By routing traffic through sidecar proxies, the mesh ensures that no single service becomes a bottleneck. However, the mesh itself can become a sunfish if misconfigured. For instance, if you configure a single control plane component (like Istiod) to handle all certificate signing and policy enforcement, it becomes a sunfish. The Linkerd documentation on retries and timeouts explicitly warns against centralized retry logic, recommending that retries be handled at the edge or by individual services.

Another real-world example comes from the open-source project "Sunfish" (a minimal chess engine). Which ironically demonstrates the efficiency of a single-purpose service. The chess engine is small, focused, and does one thing well. In contrast, the sunfish anti-pattern in distributed systems is about services that try to do too much. The lesson is clear: design services with a single responsibility, and avoid creating centralized hubs that handle cross-cutting concerns.

How to Identify a Sunfish Service in Your Infrastructure

Identifying a sunfish service requires a combination of observability data and architectural analysis. Start by looking at your service mesh metrics. In Prometheus, query for the top services by request volume and by number of unique callers. If one service has a significantly higher fan-in than others, it's a candidate. I use a simple heuristic: if a service is called by more than 10 other services and also calls more than 10 other services, it's likely a sunfish.

Another approach is to analyze your distributed traces. In Jaeger or Zipkin, look for traces where a single service appears in the critical path for multiple different workflows. For example, if the same service appears in both the "user login" trace and the "order processing" trace, it's handling unrelated concerns. This is a strong signal that the service has poor cohesion and should be split.

Finally, review your deployment topology. In Kubernetes, use tools like Kiali or Weave Scope to visualize service dependencies. A star-shaped topology with a single node in the center is a red flag. In one engagement, we used Kiali to identify a sunfish service that was handling authentication, logging. And caching for 30 microservices. The team refactored it into three separate services, reducing the blast radius by 70% and improving P99 latency by 40%.

Refactoring Strategies: Breaking the Sunfish Pattern

Once you've identified a sunfish service, the refactoring process should be incremental. The first step is to separate cross-cutting concerns into dedicated infrastructure services. For example, if the sunfish handles authentication, extract that into a dedicated Auth service with its own database and API. Use a service mesh to enforce that all internal calls go through the mesh, not through the sunfish. This is where tools like Envoy's ext_authz filter or OPA (Open Policy Agent) can help.

The second step is to introduce a proper API gateway for external traffic, and move all internal routing logic into the service mesh. This ensures that the sunfish service is only responsible for its core domain logic, not for routing or transformation. In a production migration, we used an API gateway (Kong) for external traffic and Linkerd for internal traffic. Which allowed us to gradually offload responsibilities from the sunfish service over a period of three sprints.

The third step is to add circuit breakers and bulkheads. Even after refactoring, the sunfish service may still be a dependency for some workflows. Use circuit breakers (like those provided by Resilience4j or Hystrix) to isolate failures. In Kubernetes, you can also use pod disruption budgets and horizontal pod autoscaling to ensure that the sunfish service can handle traffic spikes without collapsing. I recommend setting up alerts for when the sunfish service's error rate exceeds 1% for more than 5 minutes.

Tooling and Automation for Sunfish Detection

Automation is key to preventing the sunfish pattern from reappearing. In CI/CD pipelines, add static analysis checks that flag services with high fan-in or fan-out. Tools like kube-score or polaris can be configured to warn when a service has too many dependencies. I've also written custom PromQL queries that run as part of a pre-deployment check: if a service's request volume exceeds 50% of the cluster's total traffic, the deployment is blocked until the team provides a justification.

For runtime detection, use OpenTelemetry's service graph metrics. The traces_service_graph_request_total metric in Prometheus can show you the number of requests between every pair of services. By aggregating this data, you can compute a "sunfish score" for each service. I've implemented this in a production environment using a simple Python script that runs daily and posts results to a Slack channel. The script identifies the top three services by fan-in and alerts the team if any service has a score above a threshold.

Another useful tool is the Chaos Mesh project. Which allows you to inject failures into specific services. By running chaos experiments that target the sunfish service, you can validate whether the rest of the system can tolerate its failure. In one experiment, we killed the sunfish service for 30 seconds and observed that 12 downstream services failed. This convinced the team to prioritize the refactoring effort.

The Role of Policy Engines in Preventing Sunfish Growth

Policy engines like OPA (Open Policy Agent) can be used to enforce architectural constraints that prevent sunfish services from forming. For example, you can write a policy that rejects any deployment where a service has more than 5 incoming or outgoing connections. While this might seem draconian, it forces teams to think about service boundaries early in the design phase. In practice, we found that a policy limiting fan-in to 10 services was reasonable for most microservices, with exceptions granted only for infrastructure services like the API gateway.

Another approach is to use admission controllers in Kubernetes. When a developer tries to deploy a new service that would create a sunfish pattern, the admission controller can reject the deployment and provide a detailed explanation. I've implemented this using the kyverno policy engine. Which allows you to write policies in YAML. The policy checks for services that have both a high number of endpoints and a high number of rules in the Ingress, which is a common sunfish indicator.

Finally, consider implementing a service mesh with built-in traffic management. Linkerd, for example, provides automatic mTLS, retries. And timeouts without requiring changes to application code. By moving traffic management out of the sunfish service and into the mesh, you reduce the service's responsibilities and make it easier to refactor. The Linkerd documentation on retries and timeouts is a great starting point for understanding how to configure this.

Conclusion: Design for Failure, Not for Convenience

The sunfish anti-pattern is a classic example of how short-term convenience leads to long-term pain in distributed systems. By centralizing logic into a single service, teams trade architectural resilience for development speed. But as we've seen, the cost of that trade-off is measured in production incidents - debugging hours, and lost revenue. The solution is to design services with clear boundaries, use service meshes to handle cross-cutting concerns. And enforce architectural policies through automation.

If you're currently maintaining a sunfish service, start by instrumenting it with OpenTelemetry and running chaos experiments to understand its blast radius. Then, plan an incremental refactoring that extracts authentication, logging. And caching into dedicated services. Use tools like Linkerd, OPA. And Chaos Mesh to prevent the pattern from reappearing. Remember: a sunfish might look harmless in the ocean. But in your Kubernetes cluster, it's a ticking time bomb.

For more insights on distributed systems and observability, check out our articles on service mesh best practices and Kubernetes observability with OpenTelemetry.

Frequently Asked Questions

1. What is the sunfish anti-pattern in software engineering?
The sunfish anti-pattern refers to a single service in a microservice architecture that handles an outsized proportion of inter-service communication, becoming a centralized bottleneck it's characterized by high fan-in, high fan-out, and low cohesion, often handling unrelated concerns like authentication, logging. And routing.

2. How can I detect a sunfish service in my Kubernetes cluster.
Use service mesh metrics (eg., Prometheus queries for top services by request volume and unique callers), distributed tracing (Jaeger or Zipkin) to find services that appear in multiple critical paths. And visualization tools like Kiali to spot star-shaped topologies.

3. What tools can help refactor a sunfish service?
Service meshes like Linkerd or Istio can offload traffic management. And aPI gateways (Kong, NGINX) handle external trafficPolicy engines like OPA enforce architectural constraints. Chaos Mesh helps validate failure tolerance. OpenTelemetry provides observability during the refactoring process, since

4. Can a service mesh itself become a sunfish?
Yes, if the mesh's control plane components (e, and g, Istiod) handle all certificate signing, policy enforcement. Or configuration distribution, they can become a sunfish. Best practices recommend distributing control plane responsibilities and using sidecar proxies to handle data plane traffic independently.

5, and is the sunfish pattern always bad
In small systems with fewer than 5 services, a centralized service can be pragmatic. However, as the system scales beyond 10-15 services, the sunfish pattern introduces unacceptable risks. The key is to recognize when the cost of centralization exceeds the benefit. And to refactor before it becomes a production liability.

What do you think?

Have you encountered a sunfish service in your production environment,? And how did you refactor it without breaking downstream dependencies?

Do you think service meshes like Linkerd are the best solution for preventing sunfish patterns,? Or do they introduce their own complexity that teams should consider?

What metrics or thresholds do you use in your organization to define when a service has become a sunfish and requires immediate attention?