goplt/docs/content/architecture/architecture.md

System Architecture

This document provides a comprehensive overview of the Go Platform architecture, including system components, their relationships, and how modules integrate with the core platform.

Table of Contents

• High-Level Architecture
• Layered Architecture
• Module System Architecture
• Component Relationships
• Data Flow
• Deployment Architecture

High-Level Architecture

The Go Platform follows a microservices architecture where each module is an independent service:

• Core Services: Authentication, Identity, Authorization, Audit, etc.
• Feature Services: Blog, Billing, Analytics, etc. (modules)
• Infrastructure Services: Cache, Event Bus, Scheduler, etc.

All services communicate via gRPC (primary) or HTTP (fallback), with service discovery via a service registry. Services share infrastructure (PostgreSQL, Redis, Kafka) but are independently deployable and scalable.

graph TB
    subgraph "Go Platform"
        Core[Core Kernel]
        Module1[Module 1<br/>Blog]
        Module2[Module 2<br/>Billing]
        Module3[Module N<br/>Custom]
    end
    
    subgraph "Infrastructure"
        DB[(PostgreSQL)]
        Cache[(Redis)]
        Queue[Kafka/Event Bus]
        Storage[S3/Blob Storage]
    end
    
    subgraph "External Services"
        OIDC[OIDC Provider]
        Email[Email Service]
        Sentry[Sentry]
    end
    
    Core --> DB
    Core --> Cache
    Core --> Queue
    Core --> Storage
    Core --> OIDC
    Core --> Email
    Core --> Sentry
    
    Module1 --> Core
    Module2 --> Core
    Module3 --> Core
    
    Module1 --> DB
    Module2 --> DB
    Module3 --> DB
    
    Module1 --> Queue
    Module2 --> Queue
    
    style Core fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Module1 fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff
    style Module2 fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff
    style Module3 fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff

Layered Architecture

The platform follows a hexagonal architecture with clear separation of concerns across layers.

graph TD
    subgraph "Presentation Layer"
        HTTP[HTTP/REST API]
        GraphQL[GraphQL API]
        CLI[CLI Interface]
    end
    
    subgraph "Application Layer"
        AuthMiddleware[Auth Middleware]
        AuthzMiddleware[Authorization Middleware]
        RateLimit[Rate Limiting]
        Handlers[Request Handlers]
    end
    
    subgraph "Domain Layer"
        Services[Domain Services]
        Entities[Domain Entities]
        Policies[Business Policies]
    end
    
    subgraph "Infrastructure Layer"
        Repos[Repositories]
        CacheAdapter[Cache Adapter]
        EventBus[Event Bus]
        Jobs[Scheduler/Jobs]
    end
    
    subgraph "Core Kernel"
        DI[DI Container]
        Config[Config Manager]
        Logger[Logger]
        Metrics[Metrics]
        Health[Health Checks]
    end
    
    HTTP --> AuthMiddleware
    GraphQL --> AuthMiddleware
    CLI --> AuthMiddleware
    
    AuthMiddleware --> AuthzMiddleware
    AuthzMiddleware --> RateLimit
    RateLimit --> Handlers
    
    Handlers --> Services
    Services --> Entities
    Services --> Policies
    
    Services --> Repos
    Services --> CacheAdapter
    Services --> EventBus
    Services --> Jobs
    
    Repos --> DB[(Database)]
    CacheAdapter --> Cache[(Redis)]
    EventBus --> Queue[(Kafka)]
    
    Services --> DI
    Repos --> DI
    Handlers --> DI
    
    DI --> Config
    DI --> Logger
    DI --> Metrics
    DI --> Health
    
    style Core fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Services fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style Repos fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff

Module System Architecture

Modules are the building blocks of the platform. Each module can register services, routes, permissions, and background jobs.

graph TB
    subgraph Lifecycle["Module Lifecycle"]
        Discover["1. Discover Modules"]
        Load["2. Load Module"]
        Validate["3. Validate Dependencies"]
        Init["4. Initialize Module"]
        Start["5. Start Module"]
    end
    
    subgraph Registration["Module Registration"]
        Static["Static Registration<br/>via init()"]
        Dynamic["Dynamic Loading<br/>via .so files"]
    end
    
    subgraph Components["Module Components"]
        Routes["HTTP Routes"]
        Services["Services"]
        Repos["Repositories"]
        Perms["Permissions"]
        Jobs["Background Jobs"]
        Migrations["Database Migrations"]
    end
    
    Discover --> Load
    Load --> Static
    Load --> Dynamic
    Static --> Validate
    Dynamic --> Validate
    Validate --> Init
    Init --> Routes
    Init --> Services
    Init --> Repos
    Init --> Perms
    Init --> Jobs
    Init --> Migrations
    Routes --> Start
    Services --> Start
    Repos --> Start
    Perms --> Start
    Jobs --> Start
    Migrations --> Start
    
    classDef lifecycle fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    classDef registration fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff
    classDef components fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    classDef start fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff
    
    class Discover,Load,Validate,Init lifecycle
    class Static,Dynamic registration
    class Routes,Services,Repos,Perms,Jobs,Migrations components
    class Start start

Module Initialization Sequence

sequenceDiagram
    participant Main
    participant Loader
    participant Registry
    participant Module
    participant DI
    participant Router
    participant DB
    
    Main->>Loader: LoadModules()
    Loader->>Registry: Discover modules
    Registry-->>Loader: List of modules
    
    loop For each module
        Loader->>Module: Load module
        Module->>Registry: Register(module)
        Registry->>Registry: Validate dependencies
    end
    
    Main->>Registry: GetAllModules()
    Registry-->>Main: Ordered module list
    
    Main->>DI: Create container
    
    loop For each module
        Main->>Module: Init()
        Module->>DI: Provide services
        Module->>Router: Register routes
        Module->>DB: Register migrations
    end
    
    Main->>DB: Run migrations
    Main->>Router: Start HTTP server

Component Relationships

This diagram shows how core components interact with each other and with modules.

graph TB
    subgraph "Core Kernel Components"
        ConfigMgr[Config Manager]
        LoggerService[Logger Service]
        DI[DI Container]
        ModuleLoader[Module Loader]
        HealthRegistry[Health Registry]
        MetricsRegistry[Metrics Registry]
        ErrorBus[Error Bus]
        EventBus[Event Bus]
    end
    
    subgraph "Security Components"
        AuthService[Auth Service]
        AuthzService[Authorization Service]
        TokenProvider[Token Provider]
        PermissionResolver[Permission Resolver]
        AuditService[Audit Service]
    end
    
    subgraph "Infrastructure Components"
        DBClient[Database Client]
        CacheClient[Cache Client]
        Scheduler[Scheduler]
        Notifier[Notifier]
    end
    
    subgraph "Module Components"
        ModuleRoutes[Module Routes]
        ModuleServices[Module Services]
        ModuleRepos[Module Repositories]
    end
    
    DI --> ConfigMgr
    DI --> LoggerService
    DI --> ModuleLoader
    DI --> HealthRegistry
    DI --> MetricsRegistry
    DI --> ErrorBus
    DI --> EventBus
    DI --> AuthService
    DI --> AuthzService
    DI --> DBClient
    DI --> CacheClient
    DI --> Scheduler
    DI --> Notifier
    
    AuthService --> TokenProvider
    AuthzService --> PermissionResolver
    AuthzService --> AuditService
    
    ModuleServices --> DBClient
    ModuleServices --> CacheClient
    ModuleServices --> EventBus
    ModuleServices --> AuthzService
    
    ModuleRepos --> DBClient
    ModuleRoutes --> AuthzService
    
    Scheduler --> CacheClient
    Notifier --> EventBus
    
    ErrorBus --> LoggerService
    ErrorBus --> Sentry
    
    style DI fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style AuthService fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style ModuleServices fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff

Data Flow

Request Processing Flow

sequenceDiagram
    participant Client
    participant Router
    participant AuthMW[Auth Middleware]
    participant AuthzMW[Authz Middleware]
    participant RateLimit[Rate Limiter]
    participant Handler
    participant Service
    participant Repo
    participant DB
    participant Cache
    participant EventBus
    participant Audit
    
    Client->>Router: HTTP Request
    Router->>AuthMW: Extract JWT
    AuthMW->>AuthMW: Validate token
    AuthMW->>Router: Add user to context
    
    Router->>AuthzMW: Check permissions
    AuthzMW->>AuthzMW: Resolve permissions
    AuthzMW->>Router: Authorized
    
    Router->>RateLimit: Check rate limits
    RateLimit->>Cache: Get rate limit state
    Cache-->>RateLimit: Rate limit status
    RateLimit->>Router: Within limits
    
    Router->>Handler: Process request
    Handler->>Service: Business logic
    Service->>Cache: Check cache
    Cache-->>Service: Cache miss
    
    Service->>Repo: Query data
    Repo->>DB: Execute query
    DB-->>Repo: Return data
    Repo-->>Service: Domain entity
    Service->>Cache: Store in cache
    
    Service->>EventBus: Publish event
    Service->>Audit: Record action
    
    Service-->>Handler: Response data
    Handler-->>Router: HTTP response
    Router-->>Client: JSON response

Module Event Flow

graph LR
    subgraph "Module A"
        AService[Service A]
        AHandler[Handler A]
    end
    
    subgraph "Event Bus"
        Bus[Event Bus]
    end
    
    subgraph "Module B"
        BService[Service B]
        BHandler[Handler B]
    end
    
    subgraph "Module C"
        CService[Service C]
    end
    
    AHandler --> AService
    AService -->|Publish Event| Bus
    Bus -->|Subscribe| BService
    Bus -->|Subscribe| CService
    BService --> BHandler
    CService --> CService

Deployment Architecture

Development Deployment

graph TB
    subgraph "Developer Machine"
        IDE[IDE/Editor]
        Go[Go Runtime]
        Docker[Docker]
    end
    
    subgraph "Local Services"
        App[Platform App<br/>:8080]
        DB[(PostgreSQL<br/>:5432)]
        Redis[(Redis<br/>:6379)]
        Kafka[Kafka<br/>:9092]
    end
    
    IDE --> Go
    Go --> App
    App --> DB
    App --> Redis
    App --> Kafka
    
    Docker --> DB
    Docker --> Redis
    Docker --> Kafka

Production Deployment

graph TB
    subgraph "Load Balancer"
        LB[Load Balancer<br/>HTTPS]
    end
    
    subgraph "Platform Instances"
        App1[Platform Instance 1]
        App2[Platform Instance 2]
        App3[Platform Instance N]
    end
    
    subgraph "Database Cluster"
        Primary[(PostgreSQL<br/>Primary)]
        Replica[(PostgreSQL<br/>Replica)]
    end
    
    subgraph "Cache Cluster"
        Redis1[(Redis<br/>Master)]
        Redis2[(Redis<br/>Replica)]
    end
    
    subgraph "Message Queue"
        Kafka1[Kafka Broker 1]
        Kafka2[Kafka Broker 2]
        Kafka3[Kafka Broker 3]
    end
    
    subgraph "Observability"
        Prometheus[Prometheus]
        Grafana[Grafana]
        Jaeger[Jaeger]
        Loki[Loki]
    end
    
    subgraph "External Services"
        Sentry[Sentry]
        S3[S3 Storage]
    end
    
    LB --> App1
    LB --> App2
    LB --> App3
    
    App1 --> Primary
    App2 --> Primary
    App3 --> Primary
    App1 --> Replica
    App2 --> Replica
    App3 --> Replica
    
    App1 --> Redis1
    App2 --> Redis1
    App3 --> Redis1
    
    App1 --> Kafka1
    App2 --> Kafka2
    App3 --> Kafka3
    
    App1 --> Prometheus
    App2 --> Prometheus
    App3 --> Prometheus
    
    Prometheus --> Grafana
    App1 --> Jaeger
    App2 --> Jaeger
    App3 --> Jaeger
    App1 --> Loki
    App2 --> Loki
    App3 --> Loki
    
    App1 --> Sentry
    App2 --> Sentry
    App3 --> Sentry
    
    App1 --> S3
    App2 --> S3
    App3 --> S3
    
    style LB fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Primary fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff
    style Redis1 fill:#ff6b6b,stroke:#c92a2a,stroke-width:2px,color:#fff

Core Kernel Components

The core kernel provides the foundation for all modules. Each component has specific responsibilities:

Component Responsibilities

mindmap
  root((Core Kernel))
    Configuration
      Load configs
      Environment vars
      Secret management
    Dependency Injection
      Service registration
      Lifecycle management
      Module wiring
    Logging
      Structured logs
      Request correlation
      Log levels
    Observability
      Metrics
      Tracing
      Health checks
    Security
      Authentication
      Authorization
      Audit logging
    Module System
      Module discovery
      Module loading
      Dependency resolution

Module Integration Points

Modules integrate with the core through well-defined interfaces:

graph TB
    subgraph "Core Kernel Interfaces"
        IConfig[ConfigProvider]
        ILogger[Logger]
        IAuth[Authenticator]
        IAuthz[Authorizer]
        IEventBus[EventBus]
        ICache[Cache]
        IBlobStore[BlobStore]
        IScheduler[Scheduler]
        INotifier[Notifier]
    end
    
    subgraph "Module Implementation"
        Module[Feature Module]
        ModuleServices[Module Services]
        ModuleRoutes[Module Routes]
    end
    
    Module --> IConfig
    Module --> ILogger
    ModuleServices --> IAuth
    ModuleServices --> IAuthz
    ModuleServices --> IEventBus
    ModuleServices --> ICache
    ModuleServices --> IBlobStore
    ModuleServices --> IScheduler
    ModuleServices --> INotifier
    
    ModuleRoutes --> IAuthz
    
    style IConfig fill:#4a90e2,stroke:#2e5c8a,stroke-width:2px,color:#fff
    style Module fill:#7b68ee,stroke:#5a4fcf,stroke-width:2px,color:#fff

Microservices Architecture

The platform is designed as microservices from day one, with each module being an independent service.

Service Architecture

graph TB
    subgraph "API Gateway"
        Gateway[API Gateway<br/>Routing & Auth]
    end
    
    subgraph "Core Services"
        AuthSvc[Auth Service<br/>:8081]
        IdentitySvc[Identity Service<br/>:8082]
        AuthzSvc[Authz Service<br/>:8083]
        AuditSvc[Audit Service<br/>:8084]
    end
    
    subgraph "Feature Services"
        BlogSvc[Blog Service<br/>:8091]
        BillingSvc[Billing Service<br/>:8092]
        AnalyticsSvc[Analytics Service<br/>:8093]
    end
    
    subgraph "Infrastructure"
        DB[(PostgreSQL)]
        Cache[(Redis)]
        Queue[Kafka]
        Registry[Service Registry]
    end
    
    Gateway --> AuthSvc
    Gateway --> IdentitySvc
    Gateway --> BlogSvc
    Gateway --> BillingSvc
    
    AuthSvc --> IdentitySvc
    AuthSvc --> Registry
    BlogSvc --> AuthzSvc
    BlogSvc --> IdentitySvc
    BlogSvc --> Registry
    BillingSvc --> IdentitySvc
    BillingSvc --> Registry
    
    AuthSvc --> DB
    IdentitySvc --> DB
    BlogSvc --> DB
    BillingSvc --> DB
    
    AuthSvc --> Cache
    BlogSvc --> Cache
    BillingSvc --> Cache
    
    BlogSvc --> Queue
    BillingSvc --> Queue
    AnalyticsSvc --> Queue
    
    style Gateway fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Registry fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Service Communication

All inter-service communication uses service client interfaces:

graph TB
    subgraph "Service Client Interface"
        Interface[Service Interface<br/>pkg/services/]
    end
    
    subgraph "Implementations"
        GRPC[gRPC Client<br/>Primary]
        HTTP[HTTP Client<br/>Fallback]
    end
    
    subgraph "Service Registry"
        Registry[Service Registry<br/>Discovery & Resolution]
    end
    
    Interface --> GRPC
    Interface --> HTTP
    
    Registry --> GRPC
    Registry --> HTTP
    
    style Interface fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff
    style Registry fill:#50c878,stroke:#2e7d4e,stroke-width:2px,color:#fff

Service Communication Patterns

The platform uses three communication patterns:

1. Synchronous Service Calls (via Service Clients):

   • gRPC calls (primary) - type-safe, efficient
   • HTTP/REST calls (fallback) - for external integration
   • All calls go through service client interfaces
   • Service discovery via registry

2. Asynchronous Events (via Event Bus):

   • Distributed via Kafka
   • Preferred for cross-service communication
   • Event-driven architecture for loose coupling

3. Shared Infrastructure (For state):

   • Redis for cache and distributed state
   • PostgreSQL for persistent data
   • Kafka for events

Service Registry

The service registry enables service discovery and resolution:

graph TB
    subgraph "Service Registry"
        Registry[Service Registry Interface]
        Consul[Consul Registry]
        K8s[K8s Service Discovery]
        Etcd[etcd Registry]
    end
    
    subgraph "Services"
        AuthSvc[Auth Service]
        IdentitySvc[Identity Service]
        BlogSvc[Blog Service]
    end
    
    Registry --> Consul
    Registry --> K8s
    Registry --> Etcd
    
    Consul --> AuthSvc
    K8s --> IdentitySvc
    Etcd --> BlogSvc
    
    style Registry fill:#4a90e2,stroke:#2e5c8a,stroke-width:3px,color:#fff

Scaling Strategy

Independent Service Scaling

• Scale individual services based on load
• Independent resource allocation
• Independent deployment
• Better resource utilization
• Team autonomy

Development Mode

• For local development, multiple services can run in the same process
• Services still communicate via gRPC/HTTP (no direct calls)
• Docker Compose for easy local setup
• Maintains microservices architecture even in development

Next Steps

• Module Architecture - Detailed module architecture and design
• Module Requirements - Requirements for each module
• Component Relationships - Detailed component interactions
• ADRs - Architecture Decision Records
• ADR-0029: Microservices Architecture - Microservices strategy
• ADR-0030: Service Communication - Communication patterns