goplt/docs/content/architecture/data-flow-patterns.md

Data Flow Patterns

Purpose

This document describes how data flows through the Go Platform system, covering request/response flows, event flows, cache patterns, and observability data collection.

Overview

Data flows through the platform in multiple patterns depending on the type of operation. Understanding these patterns helps in debugging, performance optimization, and system design decisions.

Key Concepts

• Request Flow: Data flow from HTTP request to response
• Event Flow: Asynchronous data flow through event bus
• Cache Flow: Data flow through caching layers
• Observability Flow: Telemetry data collection and export

Request/Response Data Flow

Standard HTTP Request Flow

Complete data flow from HTTP request to response.

graph TD
    Start[HTTP Request] --> Auth[Authentication]
    Auth -->|Valid| Authz[Authorization]
    Auth -->|Invalid| Error1[401 Response]
    
    Authz -->|Authorized| Handler[Request Handler]
    Authz -->|Unauthorized| Error2[403 Response]
    
    Handler --> Service[Domain Service]
    Service --> Cache{Cache Check}
    
    Cache -->|Hit| CacheData[Return Cached Data]
    Cache -->|Miss| Repo[Repository]
    
    Repo --> DB[(Database)]
    DB --> Repo
    Repo --> Service
    Service --> CacheStore[Update Cache]
    
    Service --> EventBus[Publish Events]
    Service --> Audit[Audit Log]
    Service --> Metrics[Update Metrics]
    
    Service --> Handler
    Handler --> Response[HTTP Response]
    CacheData --> Response
    Error1 --> Response
    Error2 --> Response
    
    Response --> Client[Client]
    
    style Auth fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    style Service fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style Cache fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff

Request Data Transformation

How request data is transformed as it flows through the system.

sequenceDiagram
    participant Client
    participant Handler
    participant Service
    participant Repo
    participant DB
    
    Client->>Handler: HTTP Request (JSON)
    Handler->>Handler: Parse JSON
    Handler->>Handler: Validate request
    Handler->>Handler: Convert to DTO
    
    Handler->>Service: Business DTO
    Service->>Service: Business logic
    Service->>Service: Domain entity
    
    Service->>Repo: Domain entity
    Repo->>Repo: Convert to DB model
    Repo->>DB: SQL query
    
    DB-->>Repo: DB result
    Repo->>Repo: Convert to domain entity
    Repo-->>Service: Domain entity
    
    Service->>Service: Business logic
    Service->>Service: Response DTO
    Service-->>Handler: Response DTO
    
    Handler->>Handler: Convert to JSON
    Handler-->>Client: HTTP Response (JSON)

Event Data Flow

Event Publishing Flow

How events are published and flow through the event bus.

graph LR
    Publisher[Event Publisher] --> Serialize[Serialize Event]
    Serialize --> Metadata[Add Metadata]
    Metadata --> EventBus[Event Bus]
    EventBus --> Topic[Kafka Topic]
    
    Topic --> Subscriber1[Subscriber 1]
    Topic --> Subscriber2[Subscriber 2]
    Topic --> SubscriberN[Subscriber N]
    
    Subscriber1 --> Process1[Process Event]
    Subscriber2 --> Process2[Process Event]
    SubscriberN --> ProcessN[Process Event]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Topic fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff

Event Data Transformation

How event data is transformed during publishing and consumption.

sequenceDiagram
    participant Publisher
    participant EventBus
    participant Kafka
    participant Subscriber
    
    Publisher->>Publisher: Domain event
    Publisher->>EventBus: Publish(event)
    EventBus->>EventBus: Serialize to JSON
    EventBus->>EventBus: Add metadata:
      - trace_id
      - user_id
      - timestamp
      - source
    
    EventBus->>Kafka: Send to topic
    Kafka-->>EventBus: Acknowledged
    
    Kafka->>Subscriber: Deliver event
    Subscriber->>Subscriber: Deserialize JSON
    Subscriber->>Subscriber: Extract metadata
    Subscriber->>Subscriber: Domain event
    Subscriber->>Subscriber: Process event

Cache Data Flow

Cache-Aside Pattern Flow

How data flows through cache using the cache-aside pattern.

graph TD
    Start[Service Request] --> Check{Cache Hit?}
    Check -->|Yes| GetCache[Get from Cache]
    Check -->|No| GetDB[Query Database]
    
    GetCache --> Deserialize[Deserialize Data]
    Deserialize --> Return[Return Data]
    
    GetDB --> DB[(Database)]
    DB --> DBData[Database Result]
    DBData --> Serialize[Serialize Data]
    Serialize --> StoreCache[Store in Cache]
    StoreCache --> Return
    
    style Check fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style StoreCache fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Cache Invalidation Flow

How cache is invalidated when data changes.

sequenceDiagram
    participant Service
    participant Repository
    participant DB
    participant Cache
    
    Service->>Repository: Update entity
    Repository->>DB: Update database
    DB-->>Repository: Update complete
    
    Repository->>Cache: Invalidate(key)
    Cache->>Cache: Remove from cache
    Cache-->>Repository: Invalidated
    
    Repository-->>Service: Update complete
    
    Note over Service,Cache: Next read will fetch from DB and cache

Cache Write-Through Pattern

How data is written through cache to database.

sequenceDiagram
    participant Service
    participant Cache
    participant Repository
    participant DB
    
    Service->>Cache: Write data
    Cache->>Cache: Store in cache
    Cache->>Repository: Write to database
    Repository->>DB: Insert/Update
    DB-->>Repository: Success
    Repository-->>Cache: Write complete
    Cache-->>Service: Data written

Observability Data Flow

Tracing Data Flow

How distributed tracing data flows through the system.

graph TD
    Request[HTTP Request] --> Trace[Start Trace]
    Trace --> Span1[HTTP Span]
    
    Span1 --> Service[Service Call]
    Service --> Span2[Service Span]
    
    Span2 --> DB[Database Query]
    DB --> Span3[DB Span]
    
    Span2 --> gRPC[gRPC Call]
    gRPC --> Span4[gRPC Span]
    
    Span3 --> Aggregate[Collect Spans]
    Span4 --> Aggregate
    Aggregate --> Export[Export to Collector]
    Export --> Collector[OpenTelemetry Collector]
    Collector --> Backend[Backend Storage]
    
    style Trace fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Aggregate fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Metrics Data Flow

How metrics are collected and exported.

sequenceDiagram
    participant Service
    participant MetricsRegistry
    participant Exporter
    participant Prometheus
    participant Grafana
    
    Service->>Service: Business operation
    Service->>MetricsRegistry: Increment counter
    Service->>MetricsRegistry: Record duration
    Service->>MetricsRegistry: Set gauge
    
    MetricsRegistry->>MetricsRegistry: Aggregate metrics
    
    Prometheus->>Exporter: Scrape metrics
    Exporter->>MetricsRegistry: Get metrics
    MetricsRegistry-->>Exporter: Metrics data
    Exporter-->>Prometheus: Prometheus format
    
    Prometheus->>Prometheus: Store metrics
    Grafana->>Prometheus: Query metrics
    Prometheus-->>Grafana: Metrics data
    Grafana->>Grafana: Render dashboard

Log Data Flow

How logs flow through the system to various sinks.

graph TD
    Service[Service] --> Logger[Logger]
    Logger --> Format[Format Log]
    Format --> Output[Output Log]
    
    Output --> Stdout[stdout]
    Output --> File[File]
    Output --> LogCollector[Log Collector]
    
    LogCollector --> Elasticsearch[Elasticsearch]
    LogCollector --> CloudLogging[Cloud Logging]
    
    Stdout --> Container[Container Logs]
    
    style Logger fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style LogCollector fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Audit Data Flow

How audit logs flow through the system.

sequenceDiagram
    participant Service
    participant AuditClient
    participant AuditService
    participant DB
    participant Archive
    
    Service->>Service: Security-sensitive action
    Service->>AuditClient: Record audit log
    AuditClient->>AuditClient: Build audit entry:
      - actor
      - action
      - target
      - metadata
      - timestamp
    
    AuditClient->>AuditService: Store audit log
    AuditService->>AuditService: Validate entry
    AuditService->>AuditService: Ensure immutability
    AuditService->>DB: Insert audit log
    DB-->>AuditService: Log stored
    
    AuditService->>Archive: Archive old logs
    Archive->>Archive: Long-term storage
    
    Note over Service,Archive: Audit logs are immutable

Cross-Service Data Flow

Inter-Service Request Flow

How data flows when services communicate via service clients.

sequenceDiagram
    participant ServiceA
    participant ServiceClient
    participant ServiceRegistry
    participant ServiceB
    participant DB
    
    ServiceA->>ServiceClient: Call service method
    ServiceClient->>ServiceRegistry: Discover service
    ServiceRegistry-->>ServiceClient: Service endpoint
    
    ServiceClient->>ServiceB: gRPC request
    ServiceB->>ServiceB: Process request
    ServiceB->>DB: Query data
    DB-->>ServiceB: Data
    ServiceB->>ServiceB: Business logic
    ServiceB-->>ServiceClient: gRPC response
    ServiceClient-->>ServiceA: Return data

Service-to-Service Event Flow

How events flow between services.

graph LR
    ServiceA[Service A] -->|Publish| EventBus[Event Bus]
    EventBus -->|Route| ServiceB[Service B]
    EventBus -->|Route| ServiceC[Service C]
    
    ServiceB -->|Publish| EventBus
    EventBus -->|Route| ServiceD[Service D]
    
    ServiceC -->|Publish| EventBus
    EventBus -->|Route| ServiceE[Service E]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff

Data Flow Patterns Summary

Request Flow Pattern

• Path: Client → HTTP → Handler → Service → Repository → Database
• Response: Database → Repository → Service → Handler → HTTP → Client
• Side Effects: Cache updates, event publishing, audit logging, metrics

Event Flow Pattern

• Path: Publisher → Event Bus → Kafka → Subscribers
• Characteristics: Asynchronous, eventual consistency, decoupled

Cache Flow Pattern

• Read: Cache → (miss) → Database → Cache
• Write: Service → Database → Cache invalidation
• Characteristics: Performance optimization, cache-aside pattern

Observability Flow Pattern

• Tracing: Service → OpenTelemetry → Collector → Backend
• Metrics: Service → Metrics Registry → Prometheus → Grafana
• Logs: Service → Logger → Collector → Storage

Integration Points

This data flow patterns document integrates with:

• System Behavior Overview: How data flows fit into system behavior
• Service Orchestration: How data flows between services
• Module Integration Patterns: How data flows through modules
• Operational Scenarios: Data flow in specific scenarios
• Component Relationships: Component-level data flow

Related Documentation

• System Behavior Overview - System-level behavior
• Service Orchestration - Service coordination
• Module Integration Patterns - Module integration
• Operational Scenarios - Operational flows
• Component Relationships - Component dependencies
• Architecture Overview - System architecture