Files

0x1d 38a251968c docs: Align documentation with true microservices architecture

Transform all documentation from modular monolith to true microservices
architecture where core services are independently deployable.

Key Changes:
- Core Kernel: Infrastructure only (no business logic)
- Core Services: Auth, Identity, Authz, Audit as separate microservices
  - Each service has own entry point (cmd/{service}/)
  - Each service has own gRPC server and database schema
  - Services register with Consul for service discovery
- API Gateway: Moved from Epic 8 to Epic 1 as core infrastructure
  - Single entry point for all external traffic
  - Handles routing, JWT validation, rate limiting, CORS
- Service Discovery: Consul as primary mechanism (ADR-0033)
- Database Pattern: Per-service connections with schema isolation

Documentation Updates:
- Updated all 9 architecture documents
- Updated 4 ADRs and created 2 new ADRs (API Gateway, Service Discovery)
- Rewrote Epic 1: Core Kernel & Infrastructure (infrastructure only)
- Rewrote Epic 2: Core Services (Auth, Identity, Authz, Audit as services)
- Updated Epic 3-8 stories for service architecture
- Updated plan.md, playbook.md, requirements.md, index.md
- Updated all epic READMEs and story files

New ADRs:
- ADR-0032: API Gateway Strategy
- ADR-0033: Service Discovery Implementation (Consul)

New Stories:
- Epic 1.7: Service Client Interfaces
- Epic 1.8: API Gateway Implementation

2025-11-06 08:54:19 +01:00

13 KiB

Raw Blame History

Data Flow Patterns

Purpose

This document describes how data flows through the Go Platform system, covering request/response flows, event flows, cache patterns, and observability data collection.

Overview

Data flows through the platform in multiple patterns depending on the type of operation. Understanding these patterns helps in debugging, performance optimization, and system design decisions.

Key Concepts

Request Flow: Data flow from HTTP request to response
Event Flow: Asynchronous data flow through event bus
Cache Flow: Data flow through caching layers
Observability Flow: Telemetry data collection and export

Request/Response Data Flow

Standard HTTP Request Flow

Complete data flow from HTTP request through API Gateway to backend service and response.

graph TD
    Start[HTTP Request] --> Gateway[API Gateway]
    Gateway --> RateLimit{Rate Limit Check}
    RateLimit -->|Allowed| Auth[Validate JWT via Auth Service]
    RateLimit -->|Exceeded| Error0[429 Too Many Requests]
    
    Auth -->|Valid| Authz[Check Permission via Authz Service]
    Auth -->|Invalid| Error1[401 Unauthorized]
    
    Authz -->|Authorized| Route[Route to Backend Service]
    Authz -->|Unauthorized| Error2[403 Forbidden]
    
    Route --> Service[Backend Service]
    Service --> Cache{Cache Check}
    
    Cache -->|Hit| CacheData[Return Cached Data]
    Cache -->|Miss| Repo[Repository]
    
    Repo --> DB[(Database)]
    DB --> Repo
    Repo --> Service
    Service --> CacheStore[Update Cache]
    
    Service --> EventBus[Publish Events]
    Service --> AuditSvc[Audit Service<br/>gRPC]
    Service --> Metrics[Update Metrics]
    
    Service --> Gateway
    Gateway --> Response[HTTP Response]
    CacheData --> Gateway
    Error0 --> Response
    Error1 --> Response
    Error2 --> Response
    
    Response --> Client[Client]
    
    style Gateway fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Auth fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    style Service fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style Cache fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff

Request Data Transformation

How request data is transformed as it flows through API Gateway to backend service.

sequenceDiagram
    participant Client
    participant Gateway
    participant BackendService
    participant Service
    participant Repo
    participant DB
    
    Client->>Gateway: HTTP Request (JSON)
    Gateway->>Gateway: Rate limiting
    Gateway->>Gateway: Validate JWT (via Auth Service)
    Gateway->>Gateway: Check permission (via Authz Service)
    Gateway->>Gateway: Route to service (via service discovery)
    Gateway->>Gateway: Forward request (gRPC/HTTP)
    
    Gateway->>BackendService: Request (gRPC/HTTP)
    BackendService->>BackendService: Parse request
    BackendService->>BackendService: Validate request
    BackendService->>BackendService: Convert to DTO
    
    BackendService->>Service: Business DTO
    Service->>Service: Business logic
    Service->>Service: Domain entity
    
    Service->>Repo: Domain entity
    Repo->>Repo: Convert to DB model
    Repo->>DB: SQL query
    
    DB-->>Repo: DB result
    Repo->>Repo: Convert to domain entity
    Repo-->>Service: Domain entity
    
    Service->>Service: Business logic
    Service->>Service: Response DTO
    Service-->>BackendService: Response DTO
    
    BackendService->>BackendService: Convert to response format
    BackendService-->>Gateway: Response (gRPC/HTTP)
    
    Gateway->>Gateway: Transform response (if needed)
    Gateway-->>Client: HTTP Response (JSON)

Event Data Flow

Event Publishing Flow

How events are published and flow through the event bus.

graph LR
    Publisher[Event Publisher] --> Serialize[Serialize Event]
    Serialize --> Metadata[Add Metadata]
    Metadata --> EventBus[Event Bus]
    EventBus --> Topic[Kafka Topic]
    
    Topic --> Subscriber1[Subscriber 1]
    Topic --> Subscriber2[Subscriber 2]
    Topic --> SubscriberN[Subscriber N]
    
    Subscriber1 --> Process1[Process Event]
    Subscriber2 --> Process2[Process Event]
    SubscriberN --> ProcessN[Process Event]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Topic fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff

Event Data Transformation

How event data is transformed during publishing and consumption.

sequenceDiagram
    participant Publisher
    participant EventBus
    participant Kafka
    participant Subscriber
    
    Publisher->>Publisher: Domain event
    Publisher->>EventBus: Publish(event)
    EventBus->>EventBus: Serialize to JSON
    EventBus->>EventBus: Add metadata:
      - trace_id
      - user_id
      - timestamp
      - source
    
    EventBus->>Kafka: Send to topic
    Kafka-->>EventBus: Acknowledged
    
    Kafka->>Subscriber: Deliver event
    Subscriber->>Subscriber: Deserialize JSON
    Subscriber->>Subscriber: Extract metadata
    Subscriber->>Subscriber: Domain event
    Subscriber->>Subscriber: Process event

Cache Data Flow

Cache-Aside Pattern Flow

How data flows through cache using the cache-aside pattern.

graph TD
    Start[Service Request] --> Check{Cache Hit?}
    Check -->|Yes| GetCache[Get from Cache]
    Check -->|No| GetDB[Query Database]
    
    GetCache --> Deserialize[Deserialize Data]
    Deserialize --> Return[Return Data]
    
    GetDB --> DB[(Database)]
    DB --> DBData[Database Result]
    DBData --> Serialize[Serialize Data]
    Serialize --> StoreCache[Store in Cache]
    StoreCache --> Return
    
    style Check fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style StoreCache fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Cache Invalidation Flow

How cache is invalidated when data changes.

sequenceDiagram
    participant Service
    participant Repository
    participant DB
    participant Cache
    
    Service->>Repository: Update entity
    Repository->>DB: Update database
    DB-->>Repository: Update complete
    
    Repository->>Cache: Invalidate(key)
    Cache->>Cache: Remove from cache
    Cache-->>Repository: Invalidated
    
    Repository-->>Service: Update complete
    
    Note over Service,Cache: Next read will fetch from DB and cache

Cache Write-Through Pattern

How data is written through cache to database.

sequenceDiagram
    participant Service
    participant Cache
    participant Repository
    participant DB
    
    Service->>Cache: Write data
    Cache->>Cache: Store in cache
    Cache->>Repository: Write to database
    Repository->>DB: Insert/Update
    DB-->>Repository: Success
    Repository-->>Cache: Write complete
    Cache-->>Service: Data written

Observability Data Flow

Tracing Data Flow

How distributed tracing data flows through the system.

graph TD
    Request[HTTP Request] --> Trace[Start Trace]
    Trace --> Span1[HTTP Span]
    
    Span1 --> Service[Service Call]
    Service --> Span2[Service Span]
    
    Span2 --> DB[Database Query]
    DB --> Span3[DB Span]
    
    Span2 --> gRPC[gRPC Call]
    gRPC --> Span4[gRPC Span]
    
    Span3 --> Aggregate[Collect Spans]
    Span4 --> Aggregate
    Aggregate --> Export[Export to Collector]
    Export --> Collector[OpenTelemetry Collector]
    Collector --> Backend[Backend Storage]
    
    style Trace fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Aggregate fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Metrics Data Flow

How metrics are collected and exported.

sequenceDiagram
    participant Service
    participant MetricsRegistry
    participant Exporter
    participant Prometheus
    participant Grafana
    
    Service->>Service: Business operation
    Service->>MetricsRegistry: Increment counter
    Service->>MetricsRegistry: Record duration
    Service->>MetricsRegistry: Set gauge
    
    MetricsRegistry->>MetricsRegistry: Aggregate metrics
    
    Prometheus->>Exporter: Scrape metrics
    Exporter->>MetricsRegistry: Get metrics
    MetricsRegistry-->>Exporter: Metrics data
    Exporter-->>Prometheus: Prometheus format
    
    Prometheus->>Prometheus: Store metrics
    Grafana->>Prometheus: Query metrics
    Prometheus-->>Grafana: Metrics data
    Grafana->>Grafana: Render dashboard

Log Data Flow

How logs flow through the system to various sinks.

graph TD
    Service[Service] --> Logger[Logger]
    Logger --> Format[Format Log]
    Format --> Output[Output Log]
    
    Output --> Stdout[stdout]
    Output --> File[File]
    Output --> LogCollector[Log Collector]
    
    LogCollector --> Elasticsearch[Elasticsearch]
    LogCollector --> CloudLogging[Cloud Logging]
    
    Stdout --> Container[Container Logs]
    
    style Logger fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style LogCollector fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Audit Data Flow

How audit logs flow through the system.

sequenceDiagram
    participant Service
    participant AuditClient
    participant AuditService
    participant DB
    participant Archive
    
    Service->>Service: Security-sensitive action
    Service->>AuditClient: Record audit log
    AuditClient->>AuditClient: Build audit entry:
      - actor
      - action
      - target
      - metadata
      - timestamp
    
    AuditClient->>AuditService: Store audit log
    AuditService->>AuditService: Validate entry
    AuditService->>AuditService: Ensure immutability
    AuditService->>DB: Insert audit log
    DB-->>AuditService: Log stored
    
    AuditService->>Archive: Archive old logs
    Archive->>Archive: Long-term storage
    
    Note over Service,Archive: Audit logs are immutable

Cross-Service Data Flow

Inter-Service Request Flow

How data flows when services communicate via service clients.

sequenceDiagram
    participant ServiceA
    participant ServiceClient
    participant ServiceRegistry
    participant ServiceB
    participant DB
    
    ServiceA->>ServiceClient: Call service method
    ServiceClient->>ServiceRegistry: Discover service
    ServiceRegistry-->>ServiceClient: Service endpoint
    
    ServiceClient->>ServiceB: gRPC request
    ServiceB->>ServiceB: Process request
    ServiceB->>DB: Query data
    DB-->>ServiceB: Data
    ServiceB->>ServiceB: Business logic
    ServiceB-->>ServiceClient: gRPC response
    ServiceClient-->>ServiceA: Return data

Service-to-Service Event Flow

How events flow between services.

graph LR
    ServiceA[Service A] -->|Publish| EventBus[Event Bus]
    EventBus -->|Route| ServiceB[Service B]
    EventBus -->|Route| ServiceC[Service C]
    
    ServiceB -->|Publish| EventBus
    EventBus -->|Route| ServiceD[Service D]
    
    ServiceC -->|Publish| EventBus
    EventBus -->|Route| ServiceE[Service E]
    
    style EventBus fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff