Files

0x1d ace9678f6c feat: reword phase to epic, update mkdocs

2025-11-05 09:28:33 +01:00

56 KiB

Raw Blame History

Go Platform Implementation Plan

"Plug‑in‑friendly SaaS/Enterprise Platform – Go Edition"

This document outlines a complete, epic-based implementation plan for building the Go platform boilerplate based on the requirements from playbook.md and playbook-golang.md.

Executive Summary

This plan breaks down the implementation into 8 epics, each with specific deliverables and acceptance criteria. The approach prioritizes building a solid foundation (core kernel) before adding feature modules and advanced capabilities.

Total Estimated Timeline: 8-12 weeks (depending on team size and parallelization)

Key Principles:

Clean/Hexagonal Architecture with clear separation between pkg/ (interfaces) and internal/ (implementations)
Dependency Injection using uber-go/fx for lifecycle management
Microservices Architecture - each module is an independent service from day one
Service Client Interfaces - all inter-service communication via gRPC/HTTP
Service Discovery - all services register and discover via service registry
Plugin-first architecture supporting both static and dynamic module loading
Security-by-Design with JWT auth, RBAC/ABAC, and audit logging
Observability via OpenTelemetry, Prometheus, and structured logging

Epic 0: Project Setup & Foundation (Week 1)

Objectives

Initialize repository structure with proper Go project layout
Implement configuration management system
Establish structured logging system
Set up CI/CD pipeline and development tooling
Bootstrap dependency injection and application entry point

Stories

0.1 Project Initialization and Repository Structure

Goal: Establish a properly structured Go project with all necessary directories, configuration files, and documentation.

Deliverables:

Initialize Go module with correct module path
Create complete directory structure (cmd/, internal/, pkg/, modules/, config/, etc.)
Add .gitignore for Go projects
Create comprehensive README.md with project overview, setup instructions, and architecture overview
Set up basic project documentation structure

Acceptance Criteria:

go mod init creates module with correct path
All directories are in place following Go best practices
.gitignore excludes build artifacts, dependencies, and IDE files
README.md provides clear project overview and setup instructions
Project structure matches architecture documentation

0.2 Configuration Management System

Goal: Implement a flexible configuration system that loads settings from YAML files, environment variables, and supports type-safe access.

Deliverables:

ConfigProvider interface in pkg/config/ for abstraction
Viper-based implementation in internal/config/ that:
- Loads config/default.yaml as baseline
- Merges environment-specific YAML files (development/production)
- Applies environment variable overrides
- Supports nested configuration keys
- Provides type-safe getters (GetString, GetInt, GetBool, etc.)
- Supports unmarshaling into structs
Configuration files (config/default.yaml, config/development.yaml, config/production.yaml)
Configuration loader with validation

Acceptance Criteria:

Configuration loads from YAML files successfully
Environment variables override YAML values
Type-safe getters work correctly
Configuration can be unmarshaled into structs
Config system is injectable via DI container
All modules can access configuration through interface

0.3 Structured Logging System

Goal: Implement a production-ready logging system with structured JSON output, request correlation, and configurable log levels.

Deliverables:

Logger interface in pkg/logger/ with methods for all log levels
Zap-based implementation in internal/logger/ with:
- Structured JSON logging for production
- Human-readable logging for development
- Configurable log levels (debug, info, warn, error)
- Request-scoped fields support
- Context-aware logging
Request ID middleware for Gin that:
- Generates unique request IDs
- Adds request ID to all logs within request context
- Returns request ID in response headers
Global logger export via pkg/logger package

Acceptance Criteria:

Logs are structured JSON in production mode
Log levels are configurable and respected
Request IDs are generated and included in all logs
Logger can be injected via DI container
All modules can use logger through interface
Request correlation works across service boundaries

0.4 CI/CD Pipeline and Development Tooling

Goal: Establish automated testing, linting, and build processes with a developer-friendly Makefile.

Deliverables:

GitHub Actions workflow (.github/workflows/ci.yml) that:
- Sets up Go environment
- Caches Go modules
- Runs linters (golangci-lint or staticcheck)
- Runs unit tests
- Builds binary artifacts
- Validates code formatting
Comprehensive Makefile with commands:
- make test - run all tests
- make lint - run linters
- make build - build platform binary
- make docker-build - build Docker image
- make clean - clean build artifacts
- make fmt - format code
- make generate - run code generation

Acceptance Criteria:

CI pipeline runs on every push and PR
All linting checks pass
Tests run successfully (even if empty initially)
Binary builds successfully
Docker image builds successfully
Makefile commands work as expected
CI pipeline fails fast on errors

0.5 Dependency Injection and Application Bootstrap

Goal: Set up dependency injection container using Uber FX and create the application entry point that initializes the platform.

Deliverables:

DI container in internal/di/container.go that:
- Initializes Uber FX container
- Registers Config and Logger providers
- Provides lifecycle management hooks
- Supports service overrides for testing
Application entry point in cmd/platform/main.go that:
- Loads configuration
- Initializes DI container
- Sets up basic application lifecycle
- Starts minimal HTTP server (placeholder for Epic 1)
- Handles graceful shutdown

Acceptance Criteria:

DI container initializes successfully
Config and Logger are provided via DI
Application starts and shuts down gracefully
Lifecycle hooks work correctly
Services can be overridden for testing
Application compiles and runs successfully

Deliverables

✅ Repository structure in place
✅ Configuration system loads YAML files and env vars
✅ Structured logging works
✅ CI pipeline runs linting and builds binary
✅ Basic DI container initialized

Acceptance Criteria

go build ./cmd/platform succeeds
go test ./... runs (even if tests are empty)
CI pipeline passes on empty commit
Config loads from config/default.yaml

Epic 1: Core Kernel & Infrastructure (Week 2-3)

Objectives

Extend DI container to support all core services
Implement database layer with Ent ORM
Build health monitoring and metrics system
Create error handling and error bus
Establish HTTP server with comprehensive middleware stack
Integrate OpenTelemetry for distributed tracing
Create service client interfaces for microservices architecture

Stories

1.1 Enhanced Dependency Injection Container

Goal: Extend the DI container to provide all core infrastructure services with proper lifecycle management.

Deliverables:

Extended internal/di/container.go with:
- Registration of all core services
- Lifecycle management via FX
- Service override support for testing
internal/di/providers.go with provider functions:
- ProvideConfig() - configuration provider
- ProvideLogger() - logger service
- ProvideDatabase() - Ent database client
- ProvideHealthCheckers() - health check registry
- ProvideMetrics() - Prometheus metrics registry
- ProvideErrorBus() - error bus service
internal/di/core_module.go exporting CoreModule fx.Option that provides all core services

Acceptance Criteria:

All core services are provided via DI container
Services are initialized in correct dependency order
Lifecycle hooks work for all services
Services can be overridden for testing
DI container compiles without errors

1.2 Database Layer with Ent ORM

Goal: Set up a complete database layer using Ent ORM with core domain entities, migrations, and connection management.

Deliverables:

Ent schema initialization and core entities:
- User entity: ID, email, password_hash, verified, created_at, updated_at
- Role entity: ID, name, description, created_at
- Permission entity: ID, name (format: "module.resource.action")
- AuditLog entity: ID, actor_id, action, target_id, metadata (JSON), timestamp
- Many-to-many relationships: role_permissions and user_roles
Generated Ent code with proper type safety
Database client in internal/infra/database/client.go:
- NewEntClient(dsn string) (*ent.Client, error)
- Connection pooling configuration (max connections, idle timeout)
- Migration runner wrapper
- Database health check integration
Database configuration in config/default.yaml with:
- Connection string (DSN)
- Connection pool settings
- Migration settings

Acceptance Criteria:

Ent schema compiles and generates code successfully
Database client connects to PostgreSQL
Core entities can be created and queried
Migrations run successfully on startup
Connection pooling is configured correctly
Database health check works
All entities have proper indexes and relationships

1.3 Health Monitoring and Metrics System

Goal: Implement comprehensive health checks and Prometheus metrics for monitoring platform health and performance.

Deliverables:

Health check system:
- HealthChecker interface in pkg/health/health.go
- Health check registry in internal/health/registry.go:
  - Register multiple health checkers
  - GET /healthz endpoint (liveness probe)
  - GET /ready endpoint (readiness probe with database check)
  - Individual component health checks
Prometheus metrics system:
- Metrics registry setup
- HTTP request duration histogram
- HTTP request counter (by method, path, status)
- Database query duration histogram (via Ent interceptor)
- Error counter (by type)
- GET /metrics endpoint (Prometheus format)
Integration with HTTP server and DI container

Acceptance Criteria:

/healthz returns 200 when service is alive
/ready checks database connectivity and returns appropriate status
/metrics exposes Prometheus metrics in correct format
All HTTP requests are measured
Database queries are instrumented
Metrics are registered in DI container
Health checks can be extended by modules

1.4 Error Handling and Error Bus

Goal: Implement centralized error handling with an error bus that captures, logs, and optionally reports all application errors.

Deliverables:

Error bus interface in pkg/errorbus/errorbus.go:
- ErrorPublisher interface with Publish(err error) method
Channel-based error bus implementation in internal/errorbus/channel_bus.go:
- Buffered channel for error publishing
- Background goroutine consumes errors
- Logs all errors with context
- Optional: Sentry integration (Epic 6 placeholder)
Panic recovery middleware:
- Recovers from panics in HTTP handlers
- Publishes panics to error bus
- Returns appropriate HTTP error responses
Integration with DI container and HTTP middleware stack

Acceptance Criteria:

Errors are captured and logged via error bus
Panics are recovered and logged
HTTP handlers return proper error responses
Error bus is injectable via DI
Error context (request ID, user ID) is preserved
Background error consumer works correctly

1.5 HTTP Server Foundation with Middleware Stack

Goal: Create a production-ready HTTP server with comprehensive middleware for security, observability, and error handling.

Deliverables:

HTTP server in internal/server/server.go:
- Gin router initialization
- Comprehensive middleware stack:
  - Request ID generator (unique per request)
  - Structured logging middleware (logs all requests)
  - Panic recovery → error bus
  - Prometheus metrics collection
  - CORS support (configurable)
  - Request timeout handling
  - Response compression
- Core route registration:
  - GET /healthz - liveness probe
  - GET /ready - readiness probe
  - GET /metrics - Prometheus metrics
FX lifecycle integration:
- HTTP server starts on OnStart hook
- Graceful shutdown on OnStop hook (drains connections)
- Port configuration from config
Integration with main application entry point

Acceptance Criteria:

HTTP server starts successfully
All middleware executes in correct order
Request IDs are generated and logged
Metrics are collected for all requests
Panics are recovered and handled
Graceful shutdown works correctly
Server is configurable via config system
CORS is configurable per environment

1.6 OpenTelemetry Distributed Tracing

Goal: Integrate OpenTelemetry for distributed tracing across the platform to enable observability in production.

1.7 Service Client Interfaces

Goal: Create service client interfaces for all core services to enable microservices communication.

Deliverables:

Service client interfaces in pkg/services/ for all core services:
- IdentityServiceClient - User and identity operations
- AuthServiceClient - Authentication operations
- AuthzServiceClient - Authorization operations
- PermissionServiceClient - Permission resolution
- AuditServiceClient - Audit logging
Service client factory in internal/services/factory.go:
- Create gRPC clients (primary)
- Create HTTP clients (fallback)
- Support service registry integration
- Handle client lifecycle and connection pooling
Configuration for service protocol selection

Acceptance Criteria:

Service client interfaces are defined for all core services
Service factory creates gRPC clients
Service factory creates HTTP clients (fallback)
Service clients are injectable via DI
Configuration supports protocol selection
All inter-service communication goes through service clients

Deliverables:

OpenTelemetry setup in internal/observability/tracer.go:
- TracerProvider initialization
- Export to stdout (development mode)
- Export to OTLP collector (production mode)
- Trace context propagation
HTTP instrumentation middleware:
- Automatic span creation for HTTP requests
- Trace context propagation via headers
- Span attributes (method, path, status code, etc.)
Database instrumentation:
- Ent interceptor for database queries
- Query spans with timing and parameters
Integration with logger (include trace ID in logs)

Acceptance Criteria:

HTTP requests create OpenTelemetry spans
Database queries are traced
Trace context propagates across service boundaries
Trace IDs are included in logs
Traces export correctly to configured backend
Tracing works in both development and production modes
Tracing has minimal performance impact

Deliverables

✅ DI container with all core services
✅ Database client with Ent schema
✅ Health and metrics endpoints functional
✅ Error bus captures and logs errors
✅ HTTP server with middleware stack
✅ Basic observability with OpenTelemetry
✅ Service client interfaces for microservices

Acceptance Criteria

GET /healthz returns 200
GET /ready checks DB connectivity
GET /metrics exposes Prometheus metrics
Panic recovery logs errors via error bus
Database migrations run on startup
HTTP requests are traced with OpenTelemetry

Epic 2: Authentication & Authorization (Week 3-4)

Objectives

Implement complete JWT-based authentication system
Build comprehensive identity management with user lifecycle
Create role-based access control (RBAC) system
Implement authorization middleware and permission checks
Add comprehensive audit logging for security compliance
Provide database seeding for initial setup

Stories

2.1 JWT Authentication System

Goal: Implement a complete JWT-based authentication system with access tokens, refresh tokens, and secure token management.

Deliverables:

Authentication interfaces in pkg/auth/auth.go:
- Authenticator interface for token generation and verification
- TokenClaims struct with user ID, roles, tenant ID, expiration
JWT implementation in internal/auth/jwt_auth.go:
- Generate short-lived access tokens (15 minutes)
- Generate long-lived refresh tokens (7 days)
- Token signature verification
- Token expiration validation
- Claims extraction
Authentication middleware in internal/auth/middleware.go:
- Extract JWT from Authorization: Bearer <token> header
- Verify token validity
- Inject authenticated user into request context
- Helper function: auth.FromContext(ctx) *User
Authentication endpoints:
- POST /api/v1/auth/login - Authenticate user and return tokens
- POST /api/v1/auth/refresh - Refresh access token using refresh token
- Password validation against stored hashes
Integration with DI container and HTTP server

Acceptance Criteria:

Users can login and receive access and refresh tokens
Access tokens expire after configured duration
Refresh tokens can be used to obtain new access tokens
Invalid tokens are rejected with appropriate errors
Authenticated user is available in request context
Login attempts are logged
Token secrets are configurable

2.2 Identity Management System

Goal: Build a complete user identity management system with registration, email verification, password management, and user CRUD operations.

Deliverables:

Identity interfaces in pkg/identity/identity.go:
- UserRepository interface for user data access
- UserService interface for user business logic
User repository implementation in internal/identity/user_repo.go:
- CRUD operations using Ent
- Password hashing (bcrypt or argon2)
- Email uniqueness validation
- User lookup by ID and email
User service implementation in internal/identity/user_service.go:
- User registration with email verification token generation
- Email verification flow
- Password reset flow (token-based, time-limited)
- Password change with old password verification
- User profile updates
User management API endpoints:
- POST /api/v1/users - Register new user
- GET /api/v1/users/:id - Get user profile (authorized)
- PUT /api/v1/users/:id - Update user profile (authorized)
- DELETE /api/v1/users/:id - Delete user (admin only)
- POST /api/v1/users/verify-email - Verify email with token
- POST /api/v1/users/reset-password - Request password reset
- POST /api/v1/users/change-password - Change password
Integration with email notification system (Epic 5)

Acceptance Criteria:

Users can register with email and password
Passwords are securely hashed
Email verification tokens are generated and validated
Password reset flow works end-to-end
Users can update their profiles
User operations require proper authentication
All user actions are audited

2.3 Role-Based Access Control (RBAC) System

Goal: Implement a complete RBAC system with permissions, role management, and authorization middleware.

Deliverables:

Permission system in pkg/perm/perm.go:
- Permission type (string format: "module.resource.action")
- Core permission constants (system, user, role permissions)
- Permission validation utilities
Permission resolver interface in pkg/perm/resolver.go:
- HasPermission(ctx, userID, perm) method
- GetUserPermissions(ctx, userID) method
Permission resolver implementation in internal/perm/in_memory_resolver.go:
- Load user roles from database
- Load role permissions from database
- Check user permissions (with caching)
- Permission inheritance via roles
Authorization interface in pkg/auth/authz.go:
- Authorizer interface with Authorize(ctx, perm) method
RBAC authorizer implementation in internal/auth/rbac_authorizer.go:
- Extract user from context
- Resolve user permissions
- Check permission against required permission
- Return authorization errors
Authorization middleware:
- RequirePermission(perm Permission) gin.HandlerFunc decorator
- Integration with route registration
- Proper error responses for unauthorized access

Acceptance Criteria:

Permissions are defined and can be checked
Users inherit permissions through roles
Authorization middleware protects routes
Unauthorized requests return 403 errors
Permission checks are cached for performance
Permission system is extensible by modules

2.4 Role Management API

Goal: Provide complete API for managing roles, assigning permissions to roles, and assigning roles to users.

Deliverables:

Role repository in internal/identity/role_repo.go:
- CRUD operations for roles
- Assign permissions to roles (many-to-many)
- Assign roles to users (many-to-many)
- List roles with permissions
- List users with roles
Role management API endpoints:
- POST /api/v1/roles - Create new role
- GET /api/v1/roles - List all roles (with pagination)
- GET /api/v1/roles/:id - Get role details with permissions
- PUT /api/v1/roles/:id - Update role
- DELETE /api/v1/roles/:id - Delete role
- POST /api/v1/roles/:id/permissions - Assign permissions to role
- DELETE /api/v1/roles/:id/permissions/:permId - Remove permission from role
- POST /api/v1/users/:id/roles - Assign roles to user
- DELETE /api/v1/users/:id/roles/:roleId - Remove role from user
Authorization on all endpoints (admin only)
Validation and error handling

Acceptance Criteria:

Admin users can create and manage roles
Permissions can be assigned to roles
Roles can be assigned to users
Role changes affect user permissions immediately
All role operations are audited
API endpoints are protected with proper permissions

2.5 Audit Logging System

Goal: Implement comprehensive audit logging that records all security-sensitive actions for compliance and security monitoring.

Deliverables:

Audit interface in pkg/audit/audit.go:
- Auditor interface with Record(ctx, action) method
- AuditAction struct with actor, action, target, metadata
Audit implementation in internal/audit/ent_auditor.go:
- Write audit logs to audit_log table
- Capture actor from request context
- Include request metadata (ID, IP, user agent, timestamp)
- Store action details and target information
- Support JSON metadata for flexible logging
Audit middleware:
- Intercept all authenticated requests
- Record action (HTTP method + path)
- Extract user and request context
- Store audit log entry
Integration with authentication endpoints:
- Log login attempts (success and failure)
- Log password changes
- Log role assignments and removals
- Log permission changes
- Log user registration
Audit log query API (admin only):
- GET /api/v1/audit-logs - Query audit logs with filters

Acceptance Criteria:

All authenticated actions are logged
Audit logs include complete context (actor, action, target, metadata)
Audit logs are immutable (no updates/deletes)
Audit logs can be queried and filtered
Audit logging has minimal performance impact
Audit logs are stored securely

2.6 Database Seeding and Initialization

Goal: Provide database seeding functionality to create initial admin user, default roles, and core permissions.

Deliverables:

Seed script in internal/seed/seed.go:
- Create default admin user (if doesn't exist)
- Create default roles (admin, user, guest)
- Assign core permissions to roles
- Set up initial role hierarchy
Seed command in cmd/seed/main.go:
- Command-line interface for seeding
- Idempotent seeding (safe to run multiple times)
- Configuration via environment variables
Integration with application startup (optional):
- Auto-seed on first startup in development
- Manual seeding in production

Acceptance Criteria:

Seed script creates admin user successfully
Default roles are created with proper permissions
Seeding is idempotent (can run multiple times safely)
Seed script can be run via CLI
Admin user can login and manage system

Deliverables

✅ JWT authentication with access/refresh tokens
✅ User CRUD with email verification
✅ Role and permission management
✅ Authorization middleware
✅ Audit logging for all actions
✅ Seed script for initial data

Acceptance Criteria

User can register and login
JWT tokens are validated on protected routes
Users without permission get 403
All actions are logged in audit table
Admin can create roles and assign permissions
Integration test: user without permission cannot access protected resource

Epic 3: Module Framework (Week 4-5)

Objectives

Design and implement complete module system interface
Build module registry with dependency resolution
Create permission code generation from module manifests
Implement module loader supporting static and dynamic loading
Add module lifecycle management and initialization
Provide CLI tooling for module management
Implement service registry and discovery for microservices

Stories

3.1 Module System Interface and Registry

Goal: Design and implement the complete module system interface with registration, dependency resolution, and lifecycle management.

Deliverables:

Module interface in pkg/module/module.go:
- IModule interface with Name(), Version(), Dependencies(), Init(), Migrations()
- Optional lifecycle hooks: OnStart() and OnStop()
Module manifest in pkg/module/manifest.go:
- Manifest struct with Name, Version, Dependencies, Permissions, Routes
- module.yaml schema definition
Module registry in internal/registry/registry.go:
- Thread-safe module map
- Register(), All(), Get() functions
- Dependency validation (check dependencies are satisfied)
- Duplicate name detection
- Version compatibility checking

Acceptance Criteria:

Modules can register via registry.Register()
Registry validates dependencies
Registry prevents duplicate registrations
Module interface is extensible

3.2 Permission Code Generation System

Goal: Create automated permission code generation from module manifests to ensure type-safe permission constants.

Deliverables:

Permission generation script in scripts/generate-permissions.go:
- Scan all modules/*/module.yaml files
- Extract permissions from manifests
- Generate pkg/perm/generated.go with Permission constants
- Support for multiple modules
Go generate integration:
- //go:generate directive in pkg/perm/perm.go
- make generate command in Makefile
- Automatic generation on build

Acceptance Criteria:

Permission constants are generated from module.yaml
Generated code is type-safe
Code generation runs automatically
Permissions follow naming convention

3.3 Module Loader and Initialization

Goal: Implement module loading (static and dynamic) with dependency resolution and automatic initialization.

Deliverables:

Module loader in internal/pluginloader/loader.go:
- Support static registration (preferred method)
- Optional: Go plugin loading (.so files)
- Module discovery from modules/*/module.yaml
Static loader in internal/pluginloader/static_loader.go:
- Import modules via side-effect imports
- Collect all registered modules
Optional plugin loader in internal/pluginloader/plugin_loader.go:
- Scan ./plugins/*.so files
- Load via plugin.Open()
- Extract and validate module symbols
Module initializer in internal/module/initializer.go:
- Collect all registered modules
- Resolve dependency order (topological sort)
- Initialize each module's Init() fx.Option
- Merge all options into main fx container
- Run migrations in dependency order
- Handle errors gracefully
FX lifecycle integration:
- Call OnStart() during app startup
- Call OnStop() during graceful shutdown

Acceptance Criteria:

Modules load in correct dependency order
Module migrations run automatically
Module initialization integrates with FX
Lifecycle hooks work correctly
Dependency resolution handles cycles

3.4 Module Management CLI Tool

Goal: Provide CLI tooling for managing modules, validating dependencies, and testing module loading.

Deliverables:

CLI tool in cmd/platformctl/main.go:
- platformctl modules list - List all loaded modules with versions
- platformctl modules validate - Validate module dependencies
- platformctl modules test <module> - Test module loading
- platformctl modules info <module> - Show module details
Makefile integration:
- make install-cli - Install CLI tool
- make cli - Build CLI tool

Acceptance Criteria:

CLI tool lists all modules
Dependency validation works
Module testing works
CLI is installable and usable

3.5 Service Registry and Discovery

Goal: Implement a service registry that enables service discovery for microservices.

Deliverables:

Service registry interface in pkg/services/registry.go:
- ServiceRegistry interface with Register, Discover, GetService methods
- ServiceInfo struct with service metadata
Consul registry in internal/services/registry/consul.go:
- Consul integration for service discovery
- Service registration and discovery
- Health status tracking
Kubernetes service discovery in internal/services/registry/kubernetes.go:
- Kubernetes service discovery
- Service health checking
Service auto-registration on startup
Configuration for registry type selection

Acceptance Criteria:

Service registry interface is defined
Consul registry works correctly
Kubernetes registry works correctly
Services are auto-registered on startup
Service discovery works
Health checking works
Registry is configurable

Deliverables

✅ Module interface and registration system
✅ Static module registry working
✅ Permission code generation tool
✅ Module loader with dependency resolution
✅ Module initialization in main app
✅ CLI tool for module management
✅ Service registry for discovery

Acceptance Criteria

Modules can register via registry.Register()
Permission constants are generated from module.yaml
Modules load in correct dependency order
Module migrations run on startup
platformctl modules list shows all modules
Integration test: load multiple modules and verify initialization

Epic 4: Sample Feature Module (Blog) (Week 5-6)

Objectives

Create a complete sample module (Blog) to demonstrate the framework
Show how to add routes, permissions, database entities, and services
Provide reference implementation for future developers

Tasks

4.1 Blog Module Structure

Create modules/blog/ directory:

modules/blog/
├── go.mod
├── module.yaml
├── internal/
│   ├── api/
│   │   └── handler.go
│   ├── domain/
│   │   ├── post.go
│   │   └── post_repo.go
│   └── service/
│       └── post_service.go
└── pkg/
    └── module.go

Initialize go.mod:

cd modules/blog
go mod init github.com/yourorg/blog

4.2 Module Manifest

Create modules/blog/module.yaml:

name: blog
version: 0.1.0
dependencies:
  - core >= 1.0.0
permissions:
  - blog.post.create
  - blog.post.read
  - blog.post.update
  - blog.post.delete
routes:
  - method: POST
    path: /api/v1/blog/posts
    permission: blog.post.create
  - method: GET
    path: /api/v1/blog/posts/:id
    permission: blog.post.read
  - method: PUT
    path: /api/v1/blog/posts/:id
    permission: blog.post.update
  - method: DELETE
    path: /api/v1/blog/posts/:id
    permission: blog.post.delete
  - method: GET
    path: /api/v1/blog/posts
    permission: blog.post.read

4.3 Blog Domain Model

Create modules/blog/internal/domain/post.go:

type Post struct {
    ID        string
    Title     string
    Content   string
    AuthorID  string
    CreatedAt time.Time
    UpdatedAt time.Time
}

Create Ent schema modules/blog/internal/ent/schema/post.go:
- Fields: title, content, author_id (FK to user)
- Indexes: author_id, created_at
Generate Ent code for blog module

4.4 Blog Repository

Create modules/blog/internal/domain/post_repo.go:

type PostRepository interface {
    Create(ctx context.Context, p *Post) (*Post, error)
    FindByID(ctx context.Context, id string) (*Post, error)
    FindByAuthor(ctx context.Context, authorID string) ([]*Post, error)
    Update(ctx context.Context, p *Post) error
    Delete(ctx context.Context, id string) error
}

Implement using Ent client (shared from core)

4.5 Blog Service

Create modules/blog/internal/service/post_service.go:
- Business logic for creating/updating posts
- Validation (title length, content requirements)
- Authorization checks (author can only update own posts)
- Integration with audit system

4.6 Blog API Handlers

Create modules/blog/internal/api/handler.go:
- POST /api/v1/blog/posts - Create post
- GET /api/v1/blog/posts/:id - Get post
- GET /api/v1/blog/posts - List posts (with pagination)
- PUT /api/v1/blog/posts/:id - Update post
- DELETE /api/v1/blog/posts/:id - Delete post

Use authorization middleware:

grp.Use(auth.RequirePermission(perm.BlogPostCreate))

4.7 Blog Module Implementation

Create modules/blog/pkg/module.go:

type BlogModule struct{}

func (b BlogModule) Name() string { return "blog" }
func (b BlogModule) Version() string { return "0.1.0" }
func (b BlogModule) Dependencies() []string { return nil }
func (b BlogModule) Init() fx.Option {
    return fx.Options(
        fx.Provide(NewPostRepo),
        fx.Provide(NewPostService),
        fx.Invoke(RegisterHandlers),
    )
}
func (b BlogModule) Migrations() []func(*ent.Client) error {
    return []func(*ent.Client) error{
        func(c *ent.Client) error {
            return c.Schema.Create(context.Background())
        },
    }
}

var Module BlogModule

func init() {
    registry.Register(Module)
}

4.8 Integration

Update main go.mod to include blog module:

replace github.com/yourorg/blog => ./modules/blog

Import blog module in cmd/platform/main.go:
```
import _ "github.com/yourorg/blog/pkg"
```
Run permission generation: make generate
Verify blog permissions are generated

4.9 Tests

Create integration test modules/blog/internal/api/handler_test.go:
- Test creating post with valid permission
- Test creating post without permission (403)
- Test updating own post vs other's post
- Test pagination
Add unit tests for service and repository

Deliverables

✅ Complete Blog module with CRUD operations
✅ Module registered and loaded by core
✅ Permissions generated and used
✅ Routes protected with authorization
✅ Database migrations run
✅ Integration tests passing

Acceptance Criteria

Blog module loads on platform startup
POST /api/v1/blog/posts requires blog.post.create permission
User can create, read, update, delete posts
Authorization enforced (users can only edit own posts)
Integration test: full CRUD flow works
Audit logs record all blog actions

Epic 5: Infrastructure Adapters (Week 6-7)

Objectives

Implement infrastructure adapters (cache, queue, blob storage, email)
Make adapters swappable via interfaces
Add scheduler/background jobs system
Implement event bus (in-process and Kafka)
Add gRPC service definitions and clients for microservices communication

Tasks

5.1 Cache (Redis)

Install github.com/redis/go-redis/v9

Create pkg/infra/cache/cache.go interface:

type Cache interface {
    Get(ctx context.Context, key string) ([]byte, error)
    Set(ctx context.Context, key string, value []byte, ttl time.Duration) error
    Delete(ctx context.Context, key string) error
}

Implement internal/infra/cache/redis_cache.go
Add Redis config to config/default.yaml
Register in DI container
Add cache middleware for selected routes (optional)

5.2 Event Bus

Create pkg/eventbus/eventbus.go interface:

type EventBus interface {
    Publish(ctx context.Context, topic string, event Event) error
    Subscribe(topic string, handler EventHandler) error
}

Implement internal/infra/bus/inprocess_bus.go:
- Channel-based in-process bus
- Used for testing and development
Implement internal/infra/bus/kafka_bus.go:
- Install github.com/segmentio/kafka-go
- Producer for publishing
- Consumer groups for subscribing
- Error handling and retries
Add Kafka config to config/default.yaml
Register bus in DI container (switchable via config)
Add core events:
- platform.user.created
- platform.user.updated
- platform.role.assigned
- platform.permission.granted

5.3 Blob Storage

Install github.com/aws/aws-sdk-go-v2/service/s3

Create pkg/infra/blob/blob.go interface:

type BlobStore interface {
    Upload(ctx context.Context, key string, data []byte) error
    Download(ctx context.Context, key string) ([]byte, error)
    Delete(ctx context.Context, key string) error
    GetSignedURL(ctx context.Context, key string, ttl time.Duration) (string, error)
}

Implement internal/infra/blob/s3_store.go
Add S3 config to config/default.yaml
Register in DI container
Add file upload endpoint: POST /api/v1/files/upload

5.4 Email Notification

Install github.com/go-mail/mail

Create pkg/notification/notification.go interface:

type Notifier interface {
    SendEmail(ctx context.Context, to, subject, body string) error
    SendSMS(ctx context.Context, to, message string) error
}

Implement internal/infra/email/smtp_notifier.go:
- SMTP configuration
- HTML email support
- Templates for common emails (verification, password reset)
Add email config to config/default.yaml
Integrate with identity service:
- Send verification email on registration
- Send password reset email
Register in DI container

5.5 Scheduler & Background Jobs

Install github.com/robfig/cron/v3 and github.com/hibiken/asynq

Create pkg/scheduler/scheduler.go interface:

type Scheduler interface {
    Cron(spec string, job JobFunc) error
    Enqueue(queue string, payload any) error
}

Implement internal/infra/scheduler/asynq_scheduler.go:
- Redis-backed job queue
- Cron jobs for periodic tasks
- Job retries and backoff
- Job status tracking
Create internal/infra/scheduler/job_registry.go:
- Register jobs from modules
- Start job processor on app startup
Add example jobs:
- Cleanup expired tokens (daily)
- Send digest emails (weekly)
Add job monitoring endpoint: GET /api/v1/jobs/status

5.6 Secret Store Integration

Create pkg/infra/secret/secret.go interface:

type SecretStore interface {
    GetSecret(ctx context.Context, key string) (string, error)
}

Implement internal/infra/secret/vault_store.go (HashiCorp Vault):
- Install github.com/hashicorp/vault/api
- Support KV v2 secrets
Implement internal/infra/secret/aws_secrets.go (AWS Secrets Manager):
- Install github.com/aws/aws-sdk-go-v2/service/secretsmanager
Integrate with config loader:
- Overlay secrets on top of file/env config
- Load secrets lazily (cache)
Register in DI container (optional, via config)

5.7 gRPC Service Definitions and Clients

Goal: Implement gRPC service definitions and clients to enable microservices communication, allowing modules to be extracted as independent services.

Deliverables:

gRPC service definitions in api/proto/:
- Protocol Buffer files for core services (identity, auth, authz, permission, audit)
- Service and message definitions
- Proper versioning
gRPC server implementations in internal/services/grpc/server/:
- Server implementations wrapping existing services
- Error handling and validation
- Request/response conversion
gRPC client implementations in internal/services/grpc/client/:
- Clients that satisfy service client interfaces
- Connection pooling and retry logic
- Circuit breaker support
- Timeout handling
gRPC server setup and integration
Code generation from protobuf files
Configuration for enabling gRPC services

Acceptance Criteria:

gRPC service definitions are created
gRPC servers are implemented
gRPC clients implement service interfaces
Service factory can create gRPC clients
gRPC services can be enabled via configuration
Code generation works

Deliverables

✅ Cache adapter (Redis) working
✅ Event bus (in-process and Kafka) functional
✅ Blob storage (S3) adapter
✅ Email notification system
✅ Scheduler and background jobs
✅ Secret store integration (optional)
✅ gRPC service definitions and clients

Acceptance Criteria

Cache stores and retrieves data correctly
Events are published and consumed
Files can be uploaded and downloaded
Email notifications are sent
Background jobs run on schedule
Integration test: full infrastructure stack works

Epic 6: Observability & Production Readiness (Week 7-8)

Objectives

Enhance observability with full OpenTelemetry integration
Add comprehensive error reporting (Sentry)
Create Grafana dashboards
Improve logging with request correlation
Add rate limiting and security hardening

Tasks

6.1 OpenTelemetry Enhancement

Complete OpenTelemetry setup:
- Export traces to Jaeger/OTLP collector
- Add database instrumentation (Ent interceptor)
- Add Kafka instrumentation
- Add Redis instrumentation
Create custom spans:
- Module initialization spans
- Background job spans
- Event publishing spans
Add trace context propagation:
- Include trace ID in logs
- Propagate across HTTP calls
- Include in error reports

6.2 Error Reporting (Sentry)

Install github.com/getsentry/sentry-go
Integrate with error bus:
- Send errors to Sentry
- Include trace ID in Sentry events
- Add user context (user ID, email)
- Add module context (module name)
Add Sentry middleware:
- Capture panics
- Capture HTTP errors (4xx, 5xx)
Configure Sentry DSN via config

6.3 Logging Enhancements

Add request correlation:
- Generate unique request ID per request
- Include in all logs
- Return in response headers (X-Request-ID)
Add structured fields:
- user_id from context
- tenant_id from context
- module name for module logs
- trace_id from OpenTelemetry
Create log aggregation config:
- JSON format for production
- Human-readable for development
- Support for Loki/CloudWatch/ELK

6.4 Prometheus Metrics Expansion

Add more metrics:
- Database connection pool stats
- Cache hit/miss ratio
- Event bus publish/consume rates
- Background job execution times
- Module-specific metrics (via module interface)
Create metric labels:
- module label for module metrics
- tenant_id label (if multi-tenant)
- status label for error rates

6.5 Grafana Dashboards

Create ops/grafana/dashboards/:
- platform-overview.json - Overall health
- http-metrics.json - HTTP request metrics
- database-metrics.json - Database performance
- module-metrics.json - Per-module metrics
- error-rates.json - Error tracking
Document dashboard setup in docs/operations.md

6.6 Rate Limiting

Install github.com/ulule/limiter/v3
Create rate limit middleware:
- Per-user rate limiting
- Per-IP rate limiting
- Configurable limits per endpoint

Add rate limit config:

rate_limiting:
  enabled: true
  per_user: 100/minute
  per_ip: 1000/minute

Return X-RateLimit-* headers

6.7 Security Hardening

Add security headers middleware:
- X-Content-Type-Options: nosniff
- X-Frame-Options: DENY
- X-XSS-Protection: 1; mode=block
- Strict-Transport-Security (if HTTPS)
- Content-Security-Policy
Add request size limits:
- Max body size (10MB default)
- Max header size
Add input validation:
- Use github.com/go-playground/validator
- Validate all request bodies
- Sanitize user inputs
Add SQL injection protection:
- Use parameterized queries (Ent already does this)
- Add linter rule to prevent raw SQL

6.8 Performance Optimization

Add database connection pooling:
- Configure max connections
- Configure idle timeout
- Monitor pool stats
Add query optimization:
- Add indexes for common queries
- Use database query logging (development)
- Add slow query detection
Add response compression:
- Gzip middleware for large responses
Add caching strategy:
- Cache frequently accessed data (user permissions, roles)

Deliverables

✅ Full OpenTelemetry integration
✅ Sentry error reporting
✅ Enhanced logging with correlation
✅ Comprehensive Prometheus metrics
✅ Grafana dashboards
✅ Rate limiting
✅ Security hardening
✅ Performance optimizations

Acceptance Criteria

Traces are exported and visible in Jaeger
Errors are reported to Sentry with context
Logs include request IDs and trace IDs
Metrics are exposed and scraped by Prometheus
Rate limiting prevents abuse
Security headers are present
Performance meets SLA (< 100ms p95 for auth endpoints)

Epic 7: Testing, Documentation & CI/CD (Week 8-9)

Objectives

Comprehensive test coverage (unit, integration, contract)
Complete documentation
Production-ready CI/CD pipeline
Docker images and deployment guides

Tasks

7.1 Unit Tests

Achieve >80% code coverage for core modules:
- Config loader
- Logger
- Auth service
- Permission resolver
- Module registry
Use github.com/stretchr/testify for assertions
Use github.com/golang/mock or mockery for mocks
Add test helpers:
- testutil.NewTestDB() - In-memory SQLite for tests
- testutil.NewTestUser() - Create test user
- testutil.NewTestContext() - Context with user

7.2 Integration Tests

Install github.com/testcontainers/testcontainers-go
Create integration test suite:
- Full HTTP request flow
- Database operations
- Event bus publishing/consuming
- Background job execution
Test scenarios:
- User registration → login → API access
- Role assignment → permission check
- Module loading and initialization
- Multi-module interaction
Create docker-compose.test.yml:
- PostgreSQL
- Redis
- Kafka (optional)
Add test tags: //go:build integration

7.3 Contract Tests

Install github.com/pact-foundation/pact-go (optional)
Create API contract tests:
- Verify API responses match OpenAPI spec
- Test backward compatibility
Use OpenAPI validator:
- Install github.com/getkin/kin-openapi
- Validate request/response against OpenAPI spec
- Generate OpenAPI spec from code annotations

7.4 Load Testing

Create perf/ directory with k6 scripts:
- perf/auth-load.js - Login endpoint load test
- perf/api-load.js - General API load test
Document performance benchmarks:
- Request latency (p50, p95, p99)
- Throughput (requests/second)
- Resource usage (CPU, memory)

7.5 Documentation

Create README.md:
- Quick start guide
- Architecture overview
- Installation instructions
- Development setup
Create docs/architecture.md:
- System architecture diagram
- Module system explanation
- Extension points
Create docs/extension-points.md:
- How to create a module
- Permission system
- Event bus usage
- Background jobs
Create docs/api.md:
- API endpoints documentation
- Authentication flow
- Error codes
Create docs/operations.md:
- Deployment guide
- Monitoring setup
- Troubleshooting
- Grafana dashboards
Add code examples:
- examples/ directory with sample modules
- Code comments and godoc

7.6 CI/CD Pipeline Enhancement

Update .github/workflows/ci.yml:
- Run unit tests with coverage
- Run integration tests (with testcontainers)
- Run linters (golangci-lint, gosec)
- Generate coverage report
- Upload artifacts
Add release workflow:
- Semantic versioning
- Tag releases
- Build and push Docker images
- Generate changelog
Add security scanning:
- gosec for security issues
- Dependabot for dependency updates
- Trivy for container scanning

7.7 Docker Images

Create multi-stage Dockerfile:

# Build stage
FROM golang:1.22-alpine AS builder
# ... build commands

# Runtime stage
FROM gcr.io/distroless/static-debian12
# ... copy binary

Create docker-compose.yml for development:
- Platform service
- PostgreSQL
- Redis
- Kafka (optional)
Create docker-compose.prod.yml for production
Add health checks to Dockerfile
Document Docker usage in docs/deployment.md

7.8 Deployment Guides

Create docs/deployment/kubernetes.md:
- Kubernetes manifests
- Helm chart (optional)
- Service definitions
- ConfigMap and Secret management
Create docs/deployment/docker.md:
- Docker Compose deployment
- Environment variables
- Volume mounts
Create docs/deployment/cloud.md:
- AWS/GCP/Azure deployment notes
- Managed service integration
- Load balancer configuration

7.9 Developer Experience

Create Makefile with common tasks:

make dev          # Start dev environment
make test         # Run tests
make lint         # Run linters
make generate     # Generate code
make docker-build # Build Docker image
make migrate      # Run migrations

Add development scripts:
- scripts/dev.sh - Start all services
- scripts/test.sh - Run test suite
- scripts/seed.sh - Seed test data
Create .env.example with all config variables
Add pre-commit hooks (optional):
- Run linter
- Run tests
- Check formatting

Deliverables

✅ >80% test coverage
✅ Integration test suite
✅ Complete documentation
✅ Production CI/CD pipeline
✅ Docker images and deployment guides
✅ Developer tooling and scripts

Acceptance Criteria

All tests pass in CI
Code coverage >80%
Documentation is complete and accurate
Docker images build and run successfully
Deployment guides are tested
New developers can set up environment in <30 minutes

Epic 8: Advanced Features & Polish (Week 9-10, Optional)

Objectives

Add advanced features (OIDC, GraphQL, API Gateway)
Performance optimization
Additional sample modules
Final polish and bug fixes

Tasks

8.1 OpenID Connect (OIDC) Support

Install github.com/coreos/go-oidc
Implement OIDC provider:
- Discovery endpoint
- JWKS endpoint
- Token endpoint
- UserInfo endpoint
Add OIDC client support:
- Validate tokens from external IdP
- Map claims to internal user
Document OIDC setup in docs/auth.md

8.2 GraphQL API (Optional)

Install github.com/99designs/gqlgen
Create GraphQL schema:
- User queries
- Blog queries
- Mutations
Implement resolvers:
- Use existing services
- Add authorization checks
Add GraphQL endpoint: POST /graphql

8.3 API Gateway Features

Add request/response transformation
Add API key authentication
Add request routing rules
Add API versioning support

8.4 Additional Sample Modules

Create modules/notification/:
- Email templates
- Notification preferences
- Notification history
Create modules/analytics/:
- Event tracking
- Analytics dashboard API
- Export functionality

8.5 Performance Optimization

Add database query caching
Optimize N+1 queries
Add response caching (Redis)
Implement connection pooling optimizations
Add database read replicas support

8.6 Internationalization (i18n)

Install i18n library
Add locale detection:
- From Accept-Language header
- From user preferences
Create message catalogs
Add translation support for error messages

8.7 Final Polish

Code review and refactoring
Bug fixes
Performance profiling
Security audit
Documentation review

Deliverables

✅ OIDC support (optional)
✅ GraphQL API (optional)
✅ Additional sample modules
✅ Performance optimizations
✅ Final polish

Implementation Checklist Summary

Epic 0: Setup ✅

Repository structure
Configuration system
Logging foundation
Basic CI/CD
DI setup

Epic 1: Core Kernel ✅

DI container
Database (Ent)
Health & metrics
Error bus
HTTP server
OpenTelemetry

Epic 2: Auth & Authorization ✅

JWT authentication
Identity management
Roles & permissions
Authorization middleware
Audit logging

Epic 3: Module Framework ✅

Module interface
Static registry
Permission generation
Module loader
Module initialization

Epic 4: Sample Module (Blog) ✅

Blog module structure
Domain model
Repository & service
API handlers
Integration tests

Epic 5: Infrastructure ✅

Cache (Redis)
Event bus
Blob storage
Email notification
Scheduler/jobs
Multi-tenancy (optional)

Epic 6: Observability ✅

OpenTelemetry
Sentry integration
Enhanced logging
Prometheus metrics
Grafana dashboards
Rate limiting
Security hardening

Epic 7: Testing & Docs ✅

Unit tests (>80% coverage)
Integration tests
Documentation
CI/CD pipeline
Docker images
Deployment guides

Epic 8: Advanced Features (Optional) ✅

OIDC support
GraphQL API
Additional modules
Performance optimization

Risk Mitigation

Technical Risks

Risk	Impact	Mitigation
Circular import issues	High	Strict separation: interfaces in `pkg/`, implementations in `internal/`
Plugin version mismatch	Medium	Prefer static registration; document version requirements
Database migration conflicts	Medium	Central migration orchestrator, dependency ordering
Performance bottlenecks	Low	Load testing in Epic 7, profiling, caching strategy
Security vulnerabilities	High	Security audit, gosec scanning, input validation

Process Risks

Risk	Impact	Mitigation
Scope creep	Medium	Stick to epic-based approach, defer optional features to Epic 8
Incomplete documentation	Medium	Documentation as part of each epic, not afterthought
Testing gaps	High	Test coverage requirements, integration tests early

Success Criteria

The platform is considered complete when:

✅ All core modules are implemented and tested
✅ Blog module serves as working reference
✅ Test coverage >80%
✅ Documentation is complete
✅ CI/CD pipeline is production-ready
✅ Docker images build and run
✅ Integration tests pass
✅ Security audit passes
✅ Performance meets SLA (<100ms p95 for auth)
✅ New developer can set up in <30 minutes

Next Steps After Implementation

Gather Feedback: Share with team, collect requirements
Iterate: Add features based on feedback
Scale: Optimize for production load
Extend: Add more modules as needed
Community: Open source (if applicable), gather contributors

References

playbook.md - General platform playbook
playbook-golang.md - Go-specific playbook
Go Modules Documentation
Ent Documentation
Uber FX Documentation
OpenTelemetry Go

Document Version: 1.0
Status: Ready for Implementation

56 KiB Raw Blame History Unescape Escape

Go Platform Implementation Plan

Executive Summary

Epic 0: Project Setup & Foundation (Week 1)

Objectives

Stories

0.1 Project Initialization and Repository Structure

0.2 Configuration Management System

0.3 Structured Logging System

0.4 CI/CD Pipeline and Development Tooling

0.5 Dependency Injection and Application Bootstrap

Deliverables

Acceptance Criteria

Epic 1: Core Kernel & Infrastructure (Week 2-3)

Objectives

Stories

1.1 Enhanced Dependency Injection Container

1.2 Database Layer with Ent ORM

1.3 Health Monitoring and Metrics System

1.4 Error Handling and Error Bus

1.5 HTTP Server Foundation with Middleware Stack

1.6 OpenTelemetry Distributed Tracing

1.7 Service Client Interfaces

Deliverables

Acceptance Criteria

Epic 2: Authentication & Authorization (Week 3-4)

Objectives

Stories

2.1 JWT Authentication System

2.2 Identity Management System

2.3 Role-Based Access Control (RBAC) System

2.4 Role Management API

2.5 Audit Logging System

2.6 Database Seeding and Initialization

Deliverables

Acceptance Criteria

Epic 3: Module Framework (Week 4-5)

Objectives

Stories

3.1 Module System Interface and Registry

3.2 Permission Code Generation System

3.3 Module Loader and Initialization

3.4 Module Management CLI Tool

3.5 Service Registry and Discovery

Deliverables

Acceptance Criteria

Epic 4: Sample Feature Module (Blog) (Week 5-6)

Objectives

Tasks

4.1 Blog Module Structure

4.2 Module Manifest

4.3 Blog Domain Model

4.4 Blog Repository

4.5 Blog Service

4.6 Blog API Handlers

4.7 Blog Module Implementation

4.8 Integration

4.9 Tests

Deliverables

Acceptance Criteria

Epic 5: Infrastructure Adapters (Week 6-7)

Objectives

Tasks

5.1 Cache (Redis)

5.2 Event Bus

5.3 Blob Storage

5.4 Email Notification

5.5 Scheduler & Background Jobs

5.6 Secret Store Integration

5.7 gRPC Service Definitions and Clients

Deliverables

Acceptance Criteria

Epic 6: Observability & Production Readiness (Week 7-8)

Objectives

Tasks

6.1 OpenTelemetry Enhancement

6.2 Error Reporting (Sentry)

6.3 Logging Enhancements

6.4 Prometheus Metrics Expansion

6.5 Grafana Dashboards

56 KiB

Raw Blame History