118 lines
3.4 KiB
Markdown
118 lines
3.4 KiB
Markdown
# Story 1.6: OpenTelemetry Distributed Tracing
|
|
|
|
## Metadata
|
|
- **Story ID**: 1.6
|
|
- **Title**: OpenTelemetry Distributed Tracing
|
|
- **Phase**: 1 - Core Kernel & Infrastructure
|
|
- **Status**: Pending
|
|
- **Priority**: Medium
|
|
- **Estimated Time**: 5-6 hours
|
|
- **Dependencies**: 1.1, 1.5
|
|
|
|
## Goal
|
|
Integrate OpenTelemetry for distributed tracing across the platform to enable observability in production.
|
|
|
|
## Description
|
|
This story implements OpenTelemetry tracing for HTTP requests and database queries, enabling distributed tracing across the platform. Traces will be exported to stdout in development and OTLP collector in production.
|
|
|
|
## Deliverables
|
|
|
|
### 1. OpenTelemetry Setup (`internal/observability/tracer.go`)
|
|
- TracerProvider initialization
|
|
- Export to stdout (development mode)
|
|
- Export to OTLP collector (production mode)
|
|
- Trace context propagation
|
|
- Resource attributes (service name, version, etc.)
|
|
|
|
### 2. HTTP Instrumentation Middleware
|
|
- Automatic span creation for HTTP requests
|
|
- Trace context propagation via headers
|
|
- Span attributes (method, path, status code, duration)
|
|
- Error recording in spans
|
|
|
|
### 3. Database Instrumentation
|
|
- Ent interceptor for database queries
|
|
- Query spans with timing and parameters
|
|
- Database operation attributes
|
|
|
|
### 4. Integration with Logger
|
|
- Include trace ID in logs
|
|
- Correlate logs with traces
|
|
- Span context in structured logs
|
|
|
|
### 5. Configuration
|
|
- Tracing configuration in config files
|
|
- Enable/disable tracing
|
|
- Export endpoint configuration
|
|
|
|
## Implementation Steps
|
|
|
|
1. **Install Dependencies**
|
|
```bash
|
|
go get go.opentelemetry.io/otel
|
|
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
|
|
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
|
|
```
|
|
|
|
2. **Create Tracer Setup**
|
|
- Create `internal/observability/tracer.go`
|
|
- Initialize TracerProvider
|
|
- Set up exporters
|
|
|
|
3. **Add HTTP Instrumentation**
|
|
- Create HTTP middleware
|
|
- Instrument Gin router
|
|
- Add trace context propagation
|
|
|
|
4. **Add Database Instrumentation**
|
|
- Create Ent interceptor
|
|
- Instrument database queries
|
|
- Add query attributes
|
|
|
|
5. **Integrate with Logger**
|
|
- Extract trace ID from context
|
|
- Add trace ID to logs
|
|
|
|
6. **Add Configuration**
|
|
- Add tracing config
|
|
- Configure export endpoints
|
|
|
|
## Acceptance Criteria
|
|
- [ ] HTTP requests create OpenTelemetry spans
|
|
- [ ] Database queries are traced
|
|
- [ ] Trace context propagates across service boundaries
|
|
- [ ] Trace IDs are included in logs
|
|
- [ ] Traces export correctly to configured backend
|
|
- [ ] Tracing works in both development and production modes
|
|
- [ ] Tracing has minimal performance impact
|
|
- [ ] Spans have appropriate attributes
|
|
|
|
## Related ADRs
|
|
- [ADR-0016: OpenTelemetry Observability](../../adr/0016-opentelemetry-observability.md)
|
|
|
|
## Implementation Notes
|
|
- Use OpenTelemetry Go SDK
|
|
- Support both stdout and OTLP exporters
|
|
- Trace context should propagate via HTTP headers
|
|
- Consider sampling for high-volume production
|
|
- Minimize performance impact
|
|
|
|
## Testing
|
|
```bash
|
|
# Test tracing
|
|
go run cmd/platform/main.go
|
|
|
|
# Make requests and verify traces
|
|
curl http://localhost:8080/healthz
|
|
|
|
# Check trace export (stdout or OTLP)
|
|
```
|
|
|
|
## Files to Create/Modify
|
|
- `internal/observability/tracer.go` - Tracer setup
|
|
- `internal/server/middleware.go` - Add tracing middleware
|
|
- `internal/infra/database/client.go` - Add tracing interceptor
|
|
- `internal/logger/zap_logger.go` - Add trace ID to logs
|
|
- `config/default.yaml` - Add tracing configuration
|
|
|