service-mesh-observability
wshobson/agents
Implement distributed tracing, metrics, and visualization for service mesh monitoring and debugging.
What is service-mesh-observability?
Comprehensive observability for Istio, Linkerd, and service mesh deployments. Use when setting up mesh monitoring, debugging latency issues, implementing SLOs, or visualizing service dependencies and connectivity.
- Set up distributed tracing across services with span context and latency analysis
- Implement service mesh metrics collection and dashboard visualization
- Debug latency and error issues using trace correlation and dependency mapping
- Define and monitor SLOs using golden signals: latency, traffic, errors, and saturation
- Visualize service dependencies and identify bottlenecks
- Troubleshoot mesh connectivity and performance issues
How to install service-mesh-observability
npx skills add https://github.com/wshobson/agents --skill service-mesh-observabilityHow to use service-mesh-observability
- 1.Review the three pillars of observability: metrics, traces, and logs
- 2.Configure trace sampling rates appropriate for your environment (100% in dev, 1-10% in prod)
- 3.Set up trace context propagation across services using consistent headers
- 4.Implement collection for golden signals: latency (P50, P99), traffic (RPS), errors (5xx rate), saturation (resource utilization)
- 5.Create dashboards to visualize service dependencies and key metrics
- 6.Configure alerts for golden signal thresholds (e.g., P99 > 500ms, error rate > 1%)
- 7.Correlate metrics with traces using exemplars for deeper investigation
- 8.Monitor observability costs and implement hot/cold storage retention strategies
Use cases
- Setting up distributed tracing to track requests across microservices
- Implementing dashboards to monitor request rate, error rate, and latency percentiles
- Debugging high P99 latency by correlating metrics with trace spans
- Defining alert thresholds for golden signals (P99 latency > 500ms, error rate > 1%)
- Analyzing service dependencies and identifying performance bottlenecks in mesh communication
- Platform engineers setting up service mesh infrastructure
- DevOps teams implementing mesh monitoring and observability
- SREs defining SLOs and alerts for service communication
- Developers debugging latency and connectivity issues in microservices
- Architects designing observability strategies for distributed systems
service-mesh-observability FAQ
Use 100% sampling in development environments and 1-10% in production to balance visibility with storage costs. Adjust based on traffic volume and retention requirements.
Implement consistent trace context propagation by passing trace headers (like W3C Trace Context or Jaeger headers) through all service-to-service communication.
Monitor latency (P50, P99), traffic (requests per second), errors (5xx error rate), and saturation (resource utilization). Set alerts like P99 latency > 500ms and error rate > 1%.
Use exemplars to link metric data points to specific trace spans, enabling quick navigation from high-level metrics to detailed trace information for investigation.
Implement tiered retention with hot storage for recent data (7-14 days) and cold storage for older data. Monitor observability costs and adjust sampling and retention based on your budget.
Full instructions (SKILL.md)
Source of truth, from wshobson/agents.
name: service-mesh-observability description: Implement comprehensive observability for service meshes including distributed tracing, metrics, and visualization. Use when setting up mesh monitoring, debugging latency issues, or implementing SLOs for service communication.
Service Mesh Observability
Complete guide to observability patterns for Istio, Linkerd, and service mesh deployments.
When to Use This Skill
- Setting up distributed tracing across services
- Implementing service mesh metrics and dashboards
- Debugging latency and error issues
- Defining SLOs for service communication
- Visualizing service dependencies
- Troubleshooting mesh connectivity
Core Concepts
1. Three Pillars of Observability
┌─────────────────────────────────────────────────────┐
│ Observability │
├─────────────────┬─────────────────┬─────────────────┤
│ Metrics │ Traces │ Logs │
│ │ │ │
│ • Request rate │ • Span context │ • Access logs │
│ • Error rate │ • Latency │ • Error details │
│ • Latency P50 │ • Dependencies │ • Debug info │
│ • Saturation │ • Bottlenecks │ • Audit trail │
└─────────────────┴─────────────────┴─────────────────┘
2. Golden Signals for Mesh
| Signal | Description | Alert Threshold |
|---|---|---|
| Latency | Request duration P50, P99 | P99 > 500ms |
| Traffic | Requests per second | Anomaly detection |
| Errors | 5xx error rate | > 1% |
| Saturation | Resource utilization | > 80% |
Templates and detailed worked examples
Full template library and detailed worked examples live in references/details.md. Read that file when you need the concrete templates.
Best Practices
Do's
- Sample appropriately - 100% in dev, 1-10% in prod
- Use trace context - Propagate headers consistently
- Set up alerts - For golden signals
- Correlate metrics/traces - Use exemplars
- Retain strategically - Hot/cold storage tiers
Don'ts
- Don't over-sample - Storage costs add up
- Don't ignore cardinality - Limit label values
- Don't skip dashboards - Visualize dependencies
- Don't forget costs - Monitor observability costs
Related skills
More from wshobson/agents and the wider catalog.
session-guard
Monitor session health and prevent context corruption through behavioral self-enforcement.
shellcheck-configuration
Master ShellCheck static analysis configuration and usage for shell script quality.
signed-audit-trails-recipe
Cryptographically signed audit trails for Claude Code tool calls with Cedar policy enforcement.
similarity-search-patterns
Implement efficient similarity search with vector databases for semantic retrieval and nearest neighbor queries.
slo-implementation
Define and implement Service Level Indicators (SLIs), Service Level Objectives (SLOs), and error budgets for measurable reliability targets.
social-publishing
Schedule and publish posts across 13 social platforms (X, LinkedIn, Instagram, TikTok, Discord, etc.) via one API key.