goplt/docs/content/architecture/data-flow-patterns.md

# Data Flow Patterns

## Purpose

This document describes how data flows through the Go Platform system, covering request/response flows, event flows, cache patterns, and observability data collection.

## Overview

Data flows through the platform in multiple patterns depending on the type of operation. Understanding these patterns helps in debugging, performance optimization, and system design decisions.

## Key Concepts

- **Request Flow**: Data flow from HTTP request to response
- **Event Flow**: Asynchronous data flow through event bus
- **Cache Flow**: Data flow through caching layers
- **Observability Flow**: Telemetry data collection and export

## Request/Response Data Flow

### Standard HTTP Request Flow

Complete data flow from HTTP request through API Gateway to backend service and response.

```mermaid
graph TD
    Start[HTTP Request] --> Gateway[API Gateway]
    Gateway --> RateLimit{Rate Limit Check}
    RateLimit -->|Allowed| Auth[Validate JWT via Auth Service]
    RateLimit -->|Exceeded| Error0[429 Too Many Requests]
    
    Auth -->|Valid| Authz[Check Permission via Authz Service]
    Auth -->|Invalid| Error1[401 Unauthorized]
    
    Authz -->|Authorized| Route[Route to Backend Service]
    Authz -->|Unauthorized| Error2[403 Forbidden]
    
    Route --> Service[Backend Service]
    Service --> Cache{Cache Check}
    
    Cache -->|Hit| CacheData[Return Cached Data]
    Cache -->|Miss| Repo[Repository]
    
    Repo --> DB[(Database)]
    DB --> Repo
    Repo --> Service
    Service --> CacheStore[Update Cache]
    
    Service --> EventBus[Publish Events]
    Service --> AuditSvc[Audit Service<br/>gRPC]
    Service --> Metrics[Update Metrics]
    
    Service --> Gateway
    Gateway --> Response[HTTP Response]
    CacheData --> Gateway
    Error0 --> Response
    Error1 --> Response
    Error2 --> Response
    
    Response --> Client[Client]
    
    style Gateway fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Auth fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    style Service fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style Cache fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
```

### Request Data Transformation

How request data is transformed as it flows through API Gateway to backend service.

```mermaid
sequenceDiagram
    participant Client
    participant Gateway
    participant BackendService
    participant Service
    participant Repo
    participant DB
    
    Client->>Gateway: HTTP Request (JSON)
    Gateway->>Gateway: Rate limiting
    Gateway->>Gateway: Validate JWT (via Auth Service)
    Gateway->>Gateway: Check permission (via Authz Service)
    Gateway->>Gateway: Route to service (via service discovery)
    Gateway->>Gateway: Forward request (gRPC/HTTP)
    
    Gateway->>BackendService: Request (gRPC/HTTP)
    BackendService->>BackendService: Parse request
    BackendService->>BackendService: Validate request
    BackendService->>BackendService: Convert to DTO
    
    BackendService->>Service: Business DTO
    Service->>Service: Business logic
    Service->>Service: Domain entity
    
    Service->>Repo: Domain entity
    Repo->>Repo: Convert to DB model
    Repo->>DB: SQL query
    
    DB-->>Repo: DB result
    Repo->>Repo: Convert to domain entity
    Repo-->>Service: Domain entity
    
    Service->>Service: Business logic
    Service->>Service: Response DTO
    Service-->>BackendService: Response DTO
    
    BackendService->>BackendService: Convert to response format
    BackendService-->>Gateway: Response (gRPC/HTTP)
    
    Gateway->>Gateway: Transform response (if needed)
    Gateway-->>Client: HTTP Response (JSON)
```

## Event Data Flow

### Event Publishing Flow

How events are published and flow through the event bus.

```mermaid
graph LR
    Publisher[Event Publisher] --> Serialize[Serialize Event]
    Serialize --> Metadata[Add Metadata]
    Metadata --> EventBus[Event Bus]
    EventBus --> Topic[Kafka Topic]
    
    Topic --> Subscriber1[Subscriber 1]
    Topic --> Subscriber2[Subscriber 2]
    Topic --> SubscriberN[Subscriber N]
    
    Subscriber1 --> Process1[Process Event]
    Subscriber2 --> Process2[Process Event]
    SubscriberN --> ProcessN[Process Event]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Topic fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff
```

### Event Data Transformation

How event data is transformed during publishing and consumption.

```mermaid
sequenceDiagram
    participant Publisher
    participant EventBus
    participant Kafka
    participant Subscriber
    
    Publisher->>Publisher: Domain event
    Publisher->>EventBus: Publish(event)
    EventBus->>EventBus: Serialize to JSON
    EventBus->>EventBus: Add metadata:
      - trace_id
      - user_id
      - timestamp
      - source
    
    EventBus->>Kafka: Send to topic
    Kafka-->>EventBus: Acknowledged
    
    Kafka->>Subscriber: Deliver event
    Subscriber->>Subscriber: Deserialize JSON
    Subscriber->>Subscriber: Extract metadata
    Subscriber->>Subscriber: Domain event
    Subscriber->>Subscriber: Process event
```

## Cache Data Flow

### Cache-Aside Pattern Flow

How data flows through cache using the cache-aside pattern.

```mermaid
graph TD
    Start[Service Request] --> Check{Cache Hit?}
    Check -->|Yes| GetCache[Get from Cache]
    Check -->|No| GetDB[Query Database]
    
    GetCache --> Deserialize[Deserialize Data]
    Deserialize --> Return[Return Data]
    
    GetDB --> DB[(Database)]
    DB --> DBData[Database Result]
    DBData --> Serialize[Serialize Data]
    Serialize --> StoreCache[Store in Cache]
    StoreCache --> Return
    
    style Check fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style StoreCache fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
```

### Cache Invalidation Flow

How cache is invalidated when data changes.

```mermaid
sequenceDiagram
    participant Service
    participant Repository
    participant DB
    participant Cache
    
    Service->>Repository: Update entity
    Repository->>DB: Update database
    DB-->>Repository: Update complete
    
    Repository->>Cache: Invalidate(key)
    Cache->>Cache: Remove from cache
    Cache-->>Repository: Invalidated
    
    Repository-->>Service: Update complete
    
    Note over Service,Cache: Next read will fetch from DB and cache
```

### Cache Write-Through Pattern

How data is written through cache to database.

```mermaid
sequenceDiagram
    participant Service
    participant Cache
    participant Repository
    participant DB
    
    Service->>Cache: Write data
    Cache->>Cache: Store in cache
    Cache->>Repository: Write to database
    Repository->>DB: Insert/Update
    DB-->>Repository: Success
    Repository-->>Cache: Write complete
    Cache-->>Service: Data written
```

## Observability Data Flow

### Tracing Data Flow

How distributed tracing data flows through the system.

```mermaid
graph TD
    Request[HTTP Request] --> Trace[Start Trace]
    Trace --> Span1[HTTP Span]
    
    Span1 --> Service[Service Call]
    Service --> Span2[Service Span]
    
    Span2 --> DB[Database Query]
    DB --> Span3[DB Span]
    
    Span2 --> gRPC[gRPC Call]
    gRPC --> Span4[gRPC Span]
    
    Span3 --> Aggregate[Collect Spans]
    Span4 --> Aggregate
    Aggregate --> Export[Export to Collector]
    Export --> Collector[OpenTelemetry Collector]
    Collector --> Backend[Backend Storage]
    
    style Trace fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Aggregate fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
```

### Metrics Data Flow

How metrics are collected and exported.

```mermaid
sequenceDiagram
    participant Service
    participant MetricsRegistry
    participant Exporter
    participant Prometheus
    participant Grafana
    
    Service->>Service: Business operation
    Service->>MetricsRegistry: Increment counter
    Service->>MetricsRegistry: Record duration
    Service->>MetricsRegistry: Set gauge
    
    MetricsRegistry->>MetricsRegistry: Aggregate metrics
    
    Prometheus->>Exporter: Scrape metrics
    Exporter->>MetricsRegistry: Get metrics
    MetricsRegistry-->>Exporter: Metrics data
    Exporter-->>Prometheus: Prometheus format
    
    Prometheus->>Prometheus: Store metrics
    Grafana->>Prometheus: Query metrics
    Prometheus-->>Grafana: Metrics data
    Grafana->>Grafana: Render dashboard
```

### Log Data Flow

How logs flow through the system to various sinks.

```mermaid
graph TD
    Service[Service] --> Logger[Logger]
    Logger --> Format[Format Log]
    Format --> Output[Output Log]
    
    Output --> Stdout[stdout]
    Output --> File[File]
    Output --> LogCollector[Log Collector]
    
    LogCollector --> Elasticsearch[Elasticsearch]
    LogCollector --> CloudLogging[Cloud Logging]
    
    Stdout --> Container[Container Logs]
    
    style Logger fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style LogCollector fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
```

## Audit Data Flow

How audit logs flow through the system.

```mermaid
sequenceDiagram
    participant Service
    participant AuditClient
    participant AuditService
    participant DB
    participant Archive
    
    Service->>Service: Security-sensitive action
    Service->>AuditClient: Record audit log
    AuditClient->>AuditClient: Build audit entry:
      - actor
      - action
      - target
      - metadata
      - timestamp
    
    AuditClient->>AuditService: Store audit log
    AuditService->>AuditService: Validate entry
    AuditService->>AuditService: Ensure immutability
    AuditService->>DB: Insert audit log
    DB-->>AuditService: Log stored
    
    AuditService->>Archive: Archive old logs
    Archive->>Archive: Long-term storage
    
    Note over Service,Archive: Audit logs are immutable
```

## Cross-Service Data Flow

### Inter-Service Request Flow

How data flows when services communicate via service clients.

```mermaid
sequenceDiagram
    participant ServiceA
    participant ServiceClient
    participant ServiceRegistry
    participant ServiceB
    participant DB
    
    ServiceA->>ServiceClient: Call service method
    ServiceClient->>ServiceRegistry: Discover service
    ServiceRegistry-->>ServiceClient: Service endpoint
    
    ServiceClient->>ServiceB: gRPC request
    ServiceB->>ServiceB: Process request
    ServiceB->>DB: Query data
    DB-->>ServiceB: Data
    ServiceB->>ServiceB: Business logic
    ServiceB-->>ServiceClient: gRPC response
    ServiceClient-->>ServiceA: Return data
```

### Service-to-Service Event Flow

How events flow between services.

```mermaid
graph LR
    ServiceA[Service A] -->|Publish| EventBus[Event Bus]
    EventBus -->|Route| ServiceB[Service B]
    EventBus -->|Route| ServiceC[Service C]
    
    ServiceB -->|Publish| EventBus
    EventBus -->|Route| ServiceD[Service D]
    
    ServiceC -->|Publish| EventBus
    EventBus -->|Route| ServiceE[Service E]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
```

## Data Flow Patterns Summary

### Request Flow Pattern
- **Path**: Client → HTTP → Handler → Service → Repository → Database
- **Response**: Database → Repository → Service → Handler → HTTP → Client
- **Side Effects**: Cache updates, event publishing, audit logging, metrics

### Event Flow Pattern
- **Path**: Publisher → Event Bus → Kafka → Subscribers
- **Characteristics**: Asynchronous, eventual consistency, decoupled

### Cache Flow Pattern
- **Read**: Cache → (miss) → Database → Cache
- **Write**: Service → Database → Cache invalidation
- **Characteristics**: Performance optimization, cache-aside pattern

### Observability Flow Pattern
- **Tracing**: Service → OpenTelemetry → Collector → Backend
- **Metrics**: Service → Metrics Registry → Prometheus → Grafana
- **Logs**: Service → Logger → Collector → Storage

## Integration Points

This data flow patterns document integrates with:

- **[System Behavior Overview](system-behavior.md)**: How data flows fit into system behavior
- **[Service Orchestration](service-orchestration.md)**: How data flows between services
- **[Module Integration Patterns](module-integration-patterns.md)**: How data flows through modules
- **[Operational Scenarios](operational-scenarios.md)**: Data flow in specific scenarios
- **[Component Relationships](component-relationships.md)**: Component-level data flow

## Related Documentation

- [System Behavior Overview](system-behavior.md) - System-level behavior
- [Service Orchestration](service-orchestration.md) - Service coordination
- [Module Integration Patterns](module-integration-patterns.md) - Module integration
- [Operational Scenarios](operational-scenarios.md) - Operational flows
- [Component Relationships](component-relationships.md) - Component dependencies
- [Architecture Overview](architecture.md) - System architecture