goplt/docs/content/stories/epic1/1.6-opentelemetry.md

Story 1.6: OpenTelemetry Distributed Tracing

Metadata

• Story ID: 1.6
• Title: OpenTelemetry Distributed Tracing
• Epic: 1 - Core Kernel & Infrastructure
• Status: Completed
• Priority: Medium
• Estimated Time: 5-6 hours
• Dependencies: 1.1, 1.5

Goal

Integrate OpenTelemetry for distributed tracing across the platform to enable observability in production.

Description

This story implements OpenTelemetry tracing for HTTP requests and database queries, enabling distributed tracing across the platform. Traces will be exported to stdout in development and OTLP collector in production.

Deliverables

1. OpenTelemetry Setup (internal/observability/tracer.go)

• TracerProvider initialization
• Export to stdout (development mode)
• Export to OTLP collector (production mode)
• Trace context propagation
• Resource attributes (service name, version, etc.)

2. HTTP Instrumentation Middleware

• Automatic span creation for HTTP requests
• Trace context propagation via headers
• Span attributes (method, path, status code, duration)
• Error recording in spans

3. Database Instrumentation

• Ent interceptor for database queries
• Query spans with timing and parameters
• Database operation attributes

4. Integration with Logger

• Include trace ID in logs
• Correlate logs with traces
• Span context in structured logs

5. Configuration

• Tracing configuration in config files
• Enable/disable tracing
• Export endpoint configuration

Implementation Steps

1. Install Dependencies

   go get go.opentelemetry.io/otel
   go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp
   go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp
   

2. Create Tracer Setup

   • Create internal/observability/tracer.go
   • Initialize TracerProvider
   • Set up exporters

3. Add HTTP Instrumentation

   • Create HTTP middleware
   • Instrument Gin router
   • Add trace context propagation

4. Add Database Instrumentation

   • Create Ent interceptor
   • Instrument database queries
   • Add query attributes

5. Integrate with Logger

   • Extract trace ID from context
   • Add trace ID to logs

6. Add Configuration

   • Add tracing config
   • Configure export endpoints

Acceptance Criteria

• HTTP requests create OpenTelemetry spans
• Database queries are traced
• Trace context propagates across service boundaries
• Trace IDs are included in logs
• Traces export correctly to configured backend
• Tracing works in both development and production modes
• Tracing has minimal performance impact
• Spans have appropriate attributes

Related ADRs

• ADR-0016: OpenTelemetry Observability

Implementation Notes

• Use OpenTelemetry Go SDK
• Support both stdout and OTLP exporters
• Trace context should propagate via HTTP headers
• Consider sampling for high-volume production
• Minimize performance impact

Testing

# Test tracing
go run cmd/platform/main.go

# Make requests and verify traces
curl http://localhost:8080/healthz

# Check trace export (stdout or OTLP)

Files to Create/Modify

• internal/observability/tracer.go - Tracer setup
• internal/server/middleware.go - Add tracing middleware
• internal/infra/database/client.go - Add tracing interceptor
• internal/logger/zap_logger.go - Add trace ID to logs
• config/default.yaml - Add tracing configuration