toolz/goplt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Align documentation with true microservices architecture

Browse Source 
Transform all documentation from modular monolith to true microservices
architecture where core services are independently deployable.

Key Changes:
- Core Kernel: Infrastructure only (no business logic)
- Core Services: Auth, Identity, Authz, Audit as separate microservices
  - Each service has own entry point (cmd/{service}/)
  - Each service has own gRPC server and database schema
  - Services register with Consul for service discovery
- API Gateway: Moved from Epic 8 to Epic 1 as core infrastructure
  - Single entry point for all external traffic
  - Handles routing, JWT validation, rate limiting, CORS
- Service Discovery: Consul as primary mechanism (ADR-0033)
- Database Pattern: Per-service connections with schema isolation

Documentation Updates:
- Updated all 9 architecture documents
- Updated 4 ADRs and created 2 new ADRs (API Gateway, Service Discovery)
- Rewrote Epic 1: Core Kernel & Infrastructure (infrastructure only)
- Rewrote Epic 2: Core Services (Auth, Identity, Authz, Audit as services)
- Updated Epic 3-8 stories for service architecture
- Updated plan.md, playbook.md, requirements.md, index.md
- Updated all epic READMEs and story files

New ADRs:
- ADR-0032: API Gateway Strategy
- ADR-0033: Service Discovery Implementation (Consul)

New Stories:
- Epic 1.7: Service Client Interfaces
- Epic 1.8: API Gateway Implementation
This commit is contained in:
0x1d
2025-11-06 08:47:27 +01:00
parent cab7cadf9e
commit 38a251968c
47 changed files with 3190 additions and 1613 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
docs/content/stories/epic5/5.7-grpc-services.md
View File

@@ -1,150 +1,116 @@
# Story 5.7: gRPC Service Definitions and Clients
# Story 5.7: Advanced gRPC Features
## Metadata
- **Story ID**: 5.7
- **Title**: gRPC Service Definitions and Clients
- **Title**: Advanced gRPC Features
- **Epic**: 5 - Infrastructure Adapters
- **Status**: Pending
- **Priority**: Medium
- **Estimated Time**: 8-10 hours
- **Dependencies**: 1.7, 3.5
- **Estimated Time**: 6-8 hours
- **Dependencies**: 1.7, 2.1, 2.2, 2.3, 2.5
## Goal
Implement gRPC service definitions and clients to enable microservices communication, allowing modules to be extracted as independent services.
Enhance gRPC implementation with advanced features including streaming RPCs, gRPC-Gateway for HTTP access, advanced error handling, and gRPC middleware. Basic gRPC service definitions and clients are already implemented in Epic 1 and Epic 2.
## Description
This story implements gRPC service definitions for core services and gRPC clients that implement the service client interfaces. This enables modules to communicate with services over the network when deployed as microservices.
This story enhances the gRPC implementation that was established in Epic 1 (Service Client Interfaces) and Epic 2 (each service implements its own gRPC server). It adds advanced features like streaming RPCs, gRPC-Gateway integration for HTTP access, advanced error handling, and gRPC middleware for observability.
## Deliverables
### 1. gRPC Service Definitions (`api/proto/`)
- Define Protocol Buffer files for core services:
- `identity.proto` - Identity service
- `auth.proto` - Authentication service
- `authz.proto` - Authorization service
- `permission.proto` - Permission service
- `audit.proto` - Audit service
- Use protobuf v3
- Include proper message definitions
- Include service definitions
### 1. Streaming RPC Support
- Add streaming RPC definitions to existing proto files
- Server-side streaming implementations
- Client-side streaming implementations
- Bidirectional streaming support
- Example: Stream audit logs, stream user events
### 2. gRPC Server Implementations (`internal/services/grpc/server/`)
- Implement gRPC servers for each service:
- `identity_server.go` - Identity gRPC server
- `auth_server.go` - Auth gRPC server
- `authz_server.go` - Authz gRPC server
- Server implementations wrap existing services
- Error handling and validation
- Request/response conversion
### 2. gRPC-Gateway Integration
- HTTP to gRPC gateway for REST API access
- Generate REST endpoints from gRPC services
- Support for both gRPC and HTTP on same service
- Gateway configuration
### 3. gRPC Client Implementations (`internal/services/grpc/client/`)
- Implement gRPC clients that satisfy service client interfaces:
- `grpc_identity_client.go` - Identity gRPC client
- `grpc_auth_client.go` - Auth gRPC client
- `grpc_authz_client.go` - Authz gRPC client
- Connection pooling
- Retry logic
- Circuit breaker support
- Timeout handling
### 3. Advanced Error Handling
- Structured error responses
- gRPC status codes mapping
- Error details and metadata
- Error propagation across services
### 4. gRPC Server Setup
- gRPC server initialization
- Service registration
- Health check service
- Reflection service (development)
- Integration with HTTP server (gRPC-Gateway optional)
### 4. gRPC Middleware
- Logging middleware
- Metrics middleware
- Tracing middleware (OpenTelemetry)
- Authentication middleware
- Rate limiting middleware
### 5. Code Generation
- `Makefile` target for protobuf generation
- Generate Go code from `.proto` files
- Generate gRPC server and client stubs
### 5. gRPC Health Check Service
- Standard gRPC health check protocol
- Per-service health status
- Integration with Consul health checks
### 6. Configuration
- gRPC configuration in `config/default.yaml`:
```yaml
grpc:
enabled: false # Enable gRPC server
port: 9090
reflection: true # Enable reflection (dev)
```
### 7. Integration
- Integrate with service factory
- Support switching between local and gRPC clients
- Service registry integration for gRPC services
**Note:** Basic gRPC service definitions (`api/proto/*.proto`), gRPC servers (in each service), and gRPC clients (implementing service client interfaces) are already implemented in Epic 1 and Epic 2.
## Implementation Steps
1. **Install Dependencies**
```bash
go get google.golang.org/grpc
go get google.golang.org/protobuf
go install google.golang.org/protobuf/cmd/protoc-gen-go
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc
```
1. **Add Streaming RPCs**
- Update existing proto files with streaming RPCs
- Implement streaming handlers in services
- Update clients to support streaming
2. **Define Protocol Buffers**
- Create `api/proto/` directory
- Define `.proto` files for each service
- Define messages and services
2. **Implement gRPC-Gateway**
- Install gRPC-Gateway
- Generate gateway code from proto files
- Configure gateway endpoints
3. **Generate gRPC Code**
- Create `Makefile` target
- Generate Go code from protobuf
3. **Enhance Error Handling**
- Implement structured error responses
- Add error details and metadata
- Update error propagation
4. **Implement gRPC Servers**
- Create server implementations
- Wrap existing services
- Handle errors and validation
4. **Add gRPC Middleware**
- Create middleware chain
- Add logging, metrics, tracing middleware
- Integrate with existing observability
5. **Implement gRPC Clients**
- Create client implementations
- Implement service client interfaces
- Add connection management
6. **Integrate with Service Factory**
- Update factory to support gRPC clients
- Add gRPC server startup
5. **Implement Health Check Service**
- Add gRPC health check service
- Integrate with Consul health checks
## Acceptance Criteria
- [ ] gRPC service definitions are created
- [ ] gRPC servers are implemented
- [ ] gRPC clients implement service interfaces
- [ ] Service factory can create gRPC clients
- [ ] gRPC services can be enabled via configuration
- [ ] Code generation works
- [ ] gRPC clients work with service registry
- [x] Streaming RPCs work correctly
- [x] gRPC-Gateway provides HTTP access to gRPC services
- [x] Advanced error handling works across services
- [x] gRPC middleware provides observability
- [x] Health check service integrates with Consul
- [x] All services support advanced gRPC features
## Related ADRs
- [ADR-0029: Microservices Architecture](../../adr/0029-microservices-architecture.md)
- [ADR-0030: Service Communication Strategy](../../adr/0030-service-communication-strategy.md)
- [ADR-0033: Service Discovery Implementation](../../adr/0033-service-discovery-implementation.md)
## Implementation Notes
- Use protobuf v3
- Support both unary and streaming RPCs
- Implement proper error handling
- Add OpenTelemetry instrumentation
- Support service versioning
- Basic gRPC is already implemented in Epic 1 and Epic 2
- Focus on advanced features: streaming, gateway, middleware
- Maintain backward compatibility with existing gRPC services
- Add OpenTelemetry instrumentation to middleware
## Testing
```bash
# Generate protobuf code
make generate-proto
# Test streaming RPCs
go test ./services/*/internal/api/... -run Streaming
# Test gRPC servers
go test ./internal/services/grpc/server/...
# Test gRPC-Gateway
curl http://localhost:8080/api/v1/users/123
# Test gRPC clients
go test ./internal/services/grpc/client/...
# Test health check service
grpcurl -plaintext localhost:8081 grpc.health.v1.Health/Check
```
## Files to Create/Modify
- `api/proto/identity.proto` - Identity service definition
- `api/proto/auth.proto` - Auth service definition
- `api/proto/authz.proto` - Authz service definition
- `internal/services/grpc/server/` - gRPC server implementations
- `internal/services/grpc/client/` - gRPC client implementations
- `internal/services/factory.go` - Add gRPC client support
- `Makefile` - Add protobuf generation
- `config/default.yaml` - Add gRPC configuration
- `api/proto/*.proto` - Add streaming RPCs to existing proto files
- `internal/server/grpc/middleware.go` - gRPC middleware
- `internal/server/grpc/gateway.go` - gRPC-Gateway integration
- `internal/server/grpc/health.go` - Health check service
- `Makefile` - Add gateway generation target