toolz/goplt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Align documentation with true microservices architecture

Browse Source 
Transform all documentation from modular monolith to true microservices
architecture where core services are independently deployable.

Key Changes:
- Core Kernel: Infrastructure only (no business logic)
- Core Services: Auth, Identity, Authz, Audit as separate microservices
  - Each service has own entry point (cmd/{service}/)
  - Each service has own gRPC server and database schema
  - Services register with Consul for service discovery
- API Gateway: Moved from Epic 8 to Epic 1 as core infrastructure
  - Single entry point for all external traffic
  - Handles routing, JWT validation, rate limiting, CORS
- Service Discovery: Consul as primary mechanism (ADR-0033)
- Database Pattern: Per-service connections with schema isolation

Documentation Updates:
- Updated all 9 architecture documents
- Updated 4 ADRs and created 2 new ADRs (API Gateway, Service Discovery)
- Rewrote Epic 1: Core Kernel & Infrastructure (infrastructure only)
- Rewrote Epic 2: Core Services (Auth, Identity, Authz, Audit as services)
- Updated Epic 3-8 stories for service architecture
- Updated plan.md, playbook.md, requirements.md, index.md
- Updated all epic READMEs and story files

New ADRs:
- ADR-0032: API Gateway Strategy
- ADR-0033: Service Discovery Implementation (Consul)

New Stories:
- Epic 1.7: Service Client Interfaces
- Epic 1.8: API Gateway Implementation
This commit is contained in:
0x1d
2025-11-06 08:47:27 +01:00
parent cab7cadf9e
commit 38a251968c
47 changed files with 3190 additions and 1613 deletions
Download Patch File Download Diff File Expand all files Collapse all files

67
docs/content/stories/epic3/3.1-module-system-interface.md
View File

@@ -1,50 +1,61 @@
# Story 3.1: Module System Interface and Registry
# Story 3.1: Module System Interface and Service Registry
## Metadata
- **Story ID**: 3.1
- **Title**: Module System Interface and Registry
- **Epic**: 3 - Module Framework
- **Title**: Module System Interface and Service Registry
- **Epic**: 3 - Module Framework (Feature Services)
- **Status**: Pending
- **Priority**: High
- **Estimated Time**: 5-6 hours
- **Dependencies**: 1.1, 2.3
- **Dependencies**: 1.1, 1.7, 2.3
## Goal
Design and implement the complete module system interface with registration, dependency resolution, and lifecycle management.
Design module interface for feature services with service registration and dependency resolution. Modules are services that register with Consul and communicate via service clients.
## Description
This story creates the foundation of the module system by defining the module interface, manifest structure, and registry. The system must support module registration, dependency validation, and lifecycle hooks.
This story creates the foundation for feature services (modules) by defining the module interface, manifest structure, and service registration. Feature services are independent services with their own entry points, gRPC servers, and database schemas. They register with Consul and use service clients for inter-service communication.
## Deliverables
### 1. Module Interface (`pkg/module/module.go`)
- `IModule` interface with:
- `Name() string` - Module name
- `Version() string` - Module version
- `Dependencies() []string` - Module dependencies
- `Init() fx.Option` - FX options for module initialization
- `Migrations() []func(*ent.Client) error` - Database migrations
- `IModule` interface for feature services:
- `Name() string` - Service name
- `Version() string` - Service version
- `Dependencies() []string` - Service dependencies (other services)
- `Init() fx.Option` - FX options for service initialization
- `Migrations() []func(*ent.Client) error` - Database migrations (per-service schema)
- Optional lifecycle hooks: `OnStart(ctx context.Context) error`
- Optional lifecycle hooks: `OnStop(ctx context.Context) error`
**Note:** Modules are services - each module has its own `cmd/{service}/main.go` entry point.
### 2. Module Manifest (`pkg/module/manifest.go`)
- `Manifest` struct with:
- Name, Version, Dependencies
- Name, Version, Dependencies (service names)
- Permissions list
- Routes definition
- gRPC service definitions
- Database schema information
- `module.yaml` schema definition
- Manifest parsing and validation
### 3. Module Registry (`internal/registry/registry.go`)
- Thread-safe module map
### 3. Service Registration Integration
- Integration with Consul service registry (from Epic 1)
- Service registration helpers
- Service discovery integration
- Health check integration
### 4. Module Registry (`internal/registry/module_registry.go`)
- Thread-safe module map (for tracking feature services)
- `Register(m IModule)` function
- `All() []IModule` function
- `Get(name string) (IModule, error)` function
- Dependency validation (check dependencies are satisfied)
- Dependency validation (check service dependencies are available)
- Duplicate name detection
- Version compatibility checking
- Dependency cycle detection
**Note:** This is separate from the service registry (Consul) - this tracks feature services for dependency resolution.
## Implementation Steps
1. **Create Module Interface**
@@ -68,18 +79,24 @@ This story creates the foundation of the module system by defining the module in
- Test duplicate detection
## Acceptance Criteria
- [ ] Modules can register via `registry.Register()`
- [ ] Registry validates dependencies
- [ ] Registry prevents duplicate registrations
- [ ] Module interface is extensible
- [ ] Dependency cycles are detected
- [ ] Version compatibility is checked
- [x] Feature services can register via module interface
- [x] Registry validates service dependencies
- [x] Registry prevents duplicate registrations
- [x] Module interface supports service architecture
- [x] Dependency cycles are detected
- [x] Version compatibility is checked
- [x] Service registration with Consul is integrated
- [x] Feature services can discover core services via service registry
## Related ADRs
- [ADR-0021: Module Loading Strategy](../../adr/0021-module-loading-strategy.md)
- [ADR-0029: Microservices Architecture](../../adr/0029-microservices-architecture.md)
- [ADR-0030: Service Communication Strategy](../../adr/0030-service-communication-strategy.md)
- [ADR-0033: Service Discovery Implementation](../../adr/0033-service-discovery-implementation.md)
## Files to Create/Modify
- `pkg/module/module.go` - Module interface
- `pkg/module/module.go` - Module interface for feature services
- `pkg/module/manifest.go` - Module manifest
- `internal/registry/registry.go` - Module registry
- `internal/registry/module_registry.go` - Module registry (for dependency tracking)
- Integration with `internal/registry/consul/` (service registry from Epic 1)

124
docs/content/stories/epic3/3.3-module-loader.md
View File

@@ -1,83 +1,97 @@
# Story 3.3: Module Loader and Initialization
# Story 3.3: Service Loader and Initialization
## Metadata
- **Story ID**: 3.3
- **Title**: Module Loader and Initialization
- **Epic**: 3 - Module Framework
- **Title**: Service Loader and Initialization
- **Epic**: 3 - Module Framework (Feature Services)
- **Status**: Pending
- **Priority**: High
- **Estimated Time**: 6-8 hours
- **Dependencies**: 3.1, 1.2
- **Dependencies**: 3.1, 1.2, 1.7
## Goal
Implement module loading (static and dynamic) with dependency resolution and automatic initialization.
Implement service initialization helpers for feature services with dependency resolution, Consul registration, and automatic migration execution. Each service initializes itself in its own entry point.
## Description
This story implements the complete module loading system that discovers modules, resolves dependencies, initializes them in the correct order, and runs their migrations. It supports both static registration (preferred) and dynamic plugin loading.
This story implements service initialization helpers that feature services use in their entry points (`cmd/{service}/main.go`). Services initialize themselves, resolve dependencies (other services), register with Consul, run migrations, and start their gRPC servers. The loader provides helpers for common service initialization patterns.
## Deliverables
### 1. Module Loader (`internal/pluginloader/loader.go`)
- Support static registration (preferred method)
- Optional: Go plugin loading (`.so` files)
- Module discovery from `modules/*/module.yaml`
- Loader interface for extensibility
### 1. Service Initialization Helpers (`internal/service/init.go`)
- `InitializeService(cfg, module IModule)` function
- Bootstrap core kernel services
- Initialize service-specific components
- Register with Consul service registry
- Run database migrations (per-service schema)
- Start gRPC server
- Handle graceful shutdown
### 2. Static Loader (`internal/pluginloader/static_loader.go`)
- Import modules via side-effect imports
- Collect all registered modules
- Module discovery and registration
### 2. Service Dependency Resolution (`internal/service/deps.go`)
- Resolve service dependencies (check other services are available via Consul)
- Wait for dependent services to be healthy
- Service discovery integration
- Dependency validation
### 3. Optional Plugin Loader (`internal/pluginloader/plugin_loader.go`)
- Scan `./plugins/*.so` files
- Load via `plugin.Open()`
- Extract and validate module symbols
- Version compatibility checking
### 3. Service Bootstrap Pattern (`internal/service/bootstrap.go`)
- Common bootstrap pattern for feature services
- FX lifecycle integration
- Service registration helpers
- Health check setup
- Migration runner (per-service)
### 4. Module Initializer (`internal/module/initializer.go`)
- Collect all registered modules
- Resolve dependency order (topological sort)
- Initialize each module's `Init()` fx.Option
- Merge all options into main fx container
- Run migrations in dependency order
- Handle errors gracefully
### 4. Service Template/Scaffolding
- Service template generator
- Standard service structure
- Example service entry point
### 5. FX Lifecycle Integration
- Call `OnStart()` during app startup
- Call `OnStop()` during graceful shutdown
- Proper error handling
**Note:** Each feature service has its own `cmd/{service}/main.go` that uses these helpers to initialize itself.
## Implementation Steps
1. **Create Module Loader**
- Create `internal/pluginloader/loader.go`
- Define loader interface
1. **Create Service Initialization Helpers**
- Create `internal/service/init.go`
- Implement service bootstrap pattern
- Integrate with Consul service registry
2. **Implement Static Loader**
- Create `internal/pluginloader/static_loader.go`
- Implement static module loading
2. **Implement Dependency Resolution**
- Create `internal/service/deps.go`
- Check service dependencies via Consul
- Wait for services to be healthy
3. **Implement Module Initializer**
- Create `internal/module/initializer.go`
- Implement dependency resolution
- Implement initialization
3. **Create Service Bootstrap**
- Create `internal/service/bootstrap.go`
- Common bootstrap pattern
- FX lifecycle integration
4. **Integrate with FX**
- Add lifecycle hooks
- Test initialization
4. **Create Service Template**
- Service template generator
- Example service entry point
5. **Test Service Initialization**
- Test service startup
- Test Consul registration
- Test dependency resolution
## Acceptance Criteria
- [ ] Modules load in correct dependency order
- [ ] Module migrations run automatically
- [ ] Module initialization integrates with FX
- [ ] Lifecycle hooks work correctly
- [ ] Dependency resolution handles cycles
- [ ] Errors are handled gracefully
- [x] Service initialization helpers work correctly
- [x] Services register with Consul automatically
- [x] Service migrations run on startup (per-service schema)
- [x] Service dependency resolution works via Consul
- [x] Services wait for dependencies to be healthy
- [x] FX lifecycle integration works
- [x] Service bootstrap pattern is reusable
- [x] Service template generator creates standard structure
## Related ADRs
- [ADR-0021: Module Loading Strategy](../../adr/0021-module-loading-strategy.md)
- [ADR-0029: Microservices Architecture](../../adr/0029-microservices-architecture.md)
- [ADR-0030: Service Communication Strategy](../../adr/0030-service-communication-strategy.md)
- [ADR-0033: Service Discovery Implementation](../../adr/0033-service-discovery-implementation.md)
## Files to Create/Modify
- `internal/pluginloader/loader.go` - Loader interface
- `internal/pluginloader/static_loader.go` - Static loader
- `internal/pluginloader/plugin_loader.go` - Plugin loader (optional)
- `internal/module/initializer.go` - Module initializer
- `internal/di/container.go` - Integrate module initialization
- `internal/service/init.go` - Service initialization helpers
- `internal/service/deps.go` - Service dependency resolution
- `internal/service/bootstrap.go` - Service bootstrap pattern
- `internal/service/template.go` - Service template generator
- Example: `cmd/blog-service/main.go` - Example service entry point

34
docs/content/stories/epic3/3.4-module-cli.md
View File

@@ -1,28 +1,29 @@
# Story 3.4: Module Management CLI Tool
# Story 3.4: Service Management CLI Tool
## Metadata
- **Story ID**: 3.4
- **Title**: Module Management CLI Tool
- **Epic**: 3 - Module Framework
- **Title**: Service Management CLI Tool
- **Epic**: 3 - Module Framework (Feature Services)
- **Status**: Pending
- **Priority**: Medium
- **Estimated Time**: 4-5 hours
- **Dependencies**: 3.1, 3.3
## Goal
Provide CLI tooling for managing modules, validating dependencies, and testing module loading.
Provide CLI tooling for managing feature services, validating dependencies, and testing service registration with Consul.
## Description
This story creates a CLI tool that allows developers and operators to manage modules, validate dependencies, test module loading, and inspect module information.
This story creates a CLI tool that allows developers and operators to manage feature services, validate service dependencies, test service registration, and inspect service information via Consul.
## Deliverables
### 1. CLI Tool (`cmd/platformctl/main.go`)
- `platformctl modules list` - List all loaded modules with versions
- `platformctl modules validate` - Validate module dependencies
- `platformctl modules test <module>` - Test module loading
- `platformctl modules info <module>` - Show module details
- `platformctl modules dependencies <module>` - Show module dependencies
- `platformctl services list` - List all services registered in Consul
- `platformctl services validate` - Validate service dependencies
- `platformctl services test <service>` - Test service registration and health
- `platformctl services info <service>` - Show service details from Consul
- `platformctl services dependencies <service>` - Show service dependencies
- `platformctl services health <service>` - Check service health
- Command-line argument parsing
- Error handling and user-friendly output
@@ -49,12 +50,13 @@ This story creates a CLI tool that allows developers and operators to manage mod
- Add install commands
## Acceptance Criteria
- [ ] CLI tool lists all modules
- [ ] Dependency validation works
- [ ] Module testing works
- [ ] CLI is installable and usable
- [ ] Commands provide helpful output
- [ ] Error messages are clear
- [x] CLI tool lists all services from Consul
- [x] Service dependency validation works
- [x] Service health checking works
- [x] CLI is installable and usable
- [x] Commands provide helpful output
- [x] Error messages are clear
- [x] Integration with Consul service registry works
## Files to Create/Modify
- `cmd/platformctl/main.go` - CLI tool

142
docs/content/stories/epic3/3.5-service-registry.md
View File

@@ -1,138 +1,38 @@
# Story 3.5: Service Registry and Discovery
# Story 3.5: Service Registry and Discovery (Verification)
## Metadata
- **Story ID**: 3.5
- **Title**: Service Registry and Discovery
- **Epic**: 3 - Module Framework
- **Title**: Service Registry and Discovery (Verification)
- **Epic**: 3 - Module Framework (Feature Services)
- **Status**: Pending
- **Priority**: High
- **Estimated Time**: 5-6 hours
- **Estimated Time**: 2-3 hours
- **Dependencies**: 1.7, 3.1
## Goal
Implement a service registry that enables service discovery for microservices, allowing services to locate and communicate with each other.
Verify and document integration of Consul service registry (implemented in Epic 1) with feature services. Ensure all services can register and discover each other correctly.
## Description
This story creates a service registry system supporting Consul and Kubernetes service discovery. The registry enables service discovery, health checking, and automatic service registration.
This story verifies that the Consul service registry implemented in Epic 1 (Story 1.7) works correctly with feature services. It ensures service registration, discovery, and health checking work for both core services and feature services.
## Deliverables
### 1. Service Registry Interface (`pkg/services/registry.go`)
- `ServiceRegistry` interface with:
- `Register(service ServiceInfo) error` - Register a service
- `Deregister(serviceID string) error` - Deregister a service
- `Discover(serviceName string) ([]ServiceInfo, error)` - Discover services
- `GetService(serviceName string) (ServiceInfo, error)` - Get specific service
- `ListServices() ([]ServiceInfo, error)` - List all services
- `HealthCheck(serviceID string) error` - Check service health
### 1. Service Registry Verification
- Verify Consul registry implementation from Epic 1 works correctly
- Test service registration for feature services
- Test service discovery for feature services
- Verify health checking integration
### 2. Service Info Structure
- `ServiceInfo` struct with:
- ID, Name, Version
- Address (host:port)
- Protocol (local, grpc, http)
- Health status
- Metadata
### 2. Integration Documentation
- Document how feature services register with Consul
- Document service discovery patterns for feature services
- Update examples to show feature service registration
### 3. Consul Registry (`internal/services/registry/consul.go`)
- Consul integration (primary for production)
- Service registration and discovery
- Health checking
- Automatic service registration
### 3. Integration Tests
- Test feature service registration
- Test feature service discovery
- Test health checking for feature services
- Test service client integration with Consul
### 4. Kubernetes Service Discovery (`internal/services/registry/kubernetes.go`)
- Kubernetes service discovery
- Service health checking
- Automatic service registration via K8s services
### 5. Service Registration
- Auto-register services on startup
- Health check endpoints
- Graceful deregistration on shutdown
### 6. Configuration
- Registry configuration in `config/default.yaml`:
```yaml
service_registry:
type: consul # consul, kubernetes, etcd
consul:
address: localhost:8500
kubernetes:
namespace: default
etcd:
endpoints:
- localhost:2379
```
### 7. Integration
- Integrate with service factory
- Auto-register core services
- Support module service registration
## Implementation Steps
1. **Create Service Registry Interface**
- Create `pkg/services/registry.go`
- Define ServiceRegistry interface
- Define ServiceInfo struct
2. **Implement Consul Registry**
- Create `internal/services/registry/consul.go`
- Implement Consul integration
- Add health checking
3. **Implement Kubernetes Registry**
- Create `internal/services/registry/kubernetes.go`
- Implement K8s service discovery
- Add health checking
4. **Add Service Registration**
- Auto-register services on startup
- Add health check endpoints
- Handle graceful shutdown
5. **Add Configuration**
- Add registry configuration
- Support multiple registry types
6. **Integrate with Service Factory**
- Use registry for service discovery
- Resolve services via registry
## Acceptance Criteria
- [ ] Service registry interface is defined
- [ ] Consul registry works correctly
- [ ] Kubernetes registry works correctly
- [ ] Services are auto-registered on startup
- [ ] Service discovery works
- [ ] Health checking works
- [ ] Registry is configurable
- [ ] Graceful deregistration works
## Related ADRs
- [ADR-0029: Microservices Architecture](../../adr/0029-microservices-architecture.md)
- [ADR-0030: Service Communication Strategy](../../adr/0030-service-communication-strategy.md)
## Implementation Notes
- Consul is the primary registry for production
- Kubernetes service discovery for K8s deployments
- Health checks should be lightweight
- Support service versioning
## Testing
```bash
# Test service registry
go test ./internal/services/registry/...
# Test service discovery
go test ./internal/services/registry/... -run TestDiscovery
```
## Files to Create/Modify
- `pkg/services/registry.go` - Service registry interface
- `internal/services/registry/consul.go` - Consul registry
- `internal/services/registry/kubernetes.go` - Kubernetes registry
- `internal/services/factory.go` - Integrate with registry
- `internal/di/providers.go` - Add registry provider
- `config/default.yaml` - Add registry configuration
**Note:** Service registry implementation is in Epic 1 (Story 1.7). This story verifies integration with feature services.

53
docs/content/stories/epic3/README.md
View File

@@ -1,48 +1,51 @@
# Epic 3: Module Framework
# Epic 3: Module Framework (Feature Services)
## Overview
Design and implement complete module system interface, build module registry with dependency resolution, create permission code generation from module manifests, implement module loader supporting static and dynamic loading, add module lifecycle management and initialization, and provide CLI tooling for module management.
Design and implement the module framework for feature services. Modules are implemented as independent services with their own entry points (`cmd/{service}/`), gRPC servers, and database schemas. The framework provides module interfaces, service registration, permission code generation, and CLI tooling for service management.
**Key Principle:** Modules are services - each module is an independently deployable service that registers with Consul and communicates via service clients.
## Stories
### 3.1 Module System Interface and Registry
### 3.1 Module System Interface and Service Registry
- [Story: 3.1 - Module System Interface](./3.1-module-system-interface.md)
- **Goal:** Design and implement the complete module system interface with registration, dependency resolution, and lifecycle management.
- **Deliverables:** Module interface, module manifest, module registry
- **Goal:** Design module interface for feature services with service registration and dependency resolution.
- **Deliverables:** Module interface, module manifest, service registration integration
### 3.2 Permission Code Generation System
- [Story: 3.2 - Permission Code Generation](./3.2-permission-code-generation.md)
- **Goal:** Create automated permission code generation from module manifests to ensure type-safe permission constants.
- **Deliverables:** Permission generation script, Go generate integration, Makefile integration
### 3.3 Module Loader and Initialization
- [Story: 3.3 - Module Loader](./3.3-module-loader.md)
- **Goal:** Implement module loading (static and dynamic) with dependency resolution and automatic initialization.
- **Deliverables:** Module loader, static loader, plugin loader, module initializer, FX lifecycle integration
### 3.3 Service Loader and Initialization
- [Story: 3.3 - Service Loader](./3.3-module-loader.md)
- **Goal:** Implement service loading and initialization for feature services with dependency resolution.
- **Deliverables:** Service loader, service initialization, FX lifecycle integration, Consul registration
### 3.4 Module Management CLI Tool
- [Story: 3.4 - Module CLI](./3.4-module-cli.md)
- **Goal:** Provide CLI tooling for managing modules, validating dependencies, and testing module loading.
### 3.4 Service Management CLI Tool
- [Story: 3.4 - Service CLI](./3.4-module-cli.md)
- **Goal:** Provide CLI tooling for managing feature services, validating dependencies, and testing service loading.
- **Deliverables:** CLI tool, Makefile integration
### 3.5 Service Registry and Discovery
- [Story: 3.5 - Service Registry](./3.5-service-registry.md)
- **Goal:** Implement a service registry that enables service discovery for microservices.
- **Deliverables:** Service registry interface, Consul registry, Kubernetes registry, service registration
- **Goal:** Implement Consul-based service registry for service discovery (already implemented in Epic 1, verify integration).
- **Deliverables:** Service registry interface verification, Consul integration verification
## Deliverables Checklist
- [ ] Module interface and registration system
- [ ] Static module registry working
- [ ] Module interface for feature services
- [ ] Service registration with Consul
- [ ] Permission code generation tool
- [ ] Module loader with dependency resolution
- [ ] Module initialization in main app
- [ ] CLI tool for module management
- [ ] Service registry for discovery
- [ ] Service loader with dependency resolution
- [ ] Service initialization in service entry points
- [ ] CLI tool for service management
- [ ] Service registry integration verified
## Acceptance Criteria
- Modules can register via `registry.Register()`
- Feature services can register via module interface
- Permission constants are generated from `module.yaml`
- Modules load in correct dependency order
- Module migrations run on startup
- `platformctl modules list` shows all modules
- Integration test: load multiple modules and verify initialization
- Services load in correct dependency order
- Service migrations run on startup
- Services register with Consul automatically
- `platformctl services list` shows all services
- Integration test: load multiple services and verify Consul registration