spore/docs/Development.md

# Development & Deployment Guide

## Prerequisites

### Required Tools

- **PlatformIO Core** or **PlatformIO IDE**
- **ESP8266 development tools**
- **`jq`** for JSON processing in scripts
- **Git** for version control

### System Requirements

- **Operating System**: Linux, macOS, or Windows
- **Python**: 3.7+ (for PlatformIO)
- **Memory**: 4GB+ RAM recommended
- **Storage**: 2GB+ free space for development environment

## Project Structure

```
spore/
├── src/                    # Source code
│   ├── main.cpp           # Main application entry point
│   ├── ApiServer.cpp      # HTTP API server implementation
│   ├── ClusterManager.cpp # Cluster management logic
│   ├── NetworkManager.cpp # WiFi and network handling
│   ├── TaskManager.cpp    # Background task management
│   └── NodeContext.cpp    # Central context and events
├── include/                # Header files
├── lib/                    # Library files
├── docs/                   # Documentation
├── api/                    # OpenAPI specification
├── examples/               # Example code
├── test/                   # Test files
├── platformio.ini         # PlatformIO configuration
└── ctl.sh                 # Build and deployment scripts
```

## PlatformIO Configuration

### Framework and Board

The project uses PlatformIO with the following configuration:

```ini
[env:esp01_1m]
platform = platformio/espressif8266@^4.2.1
board = esp01_1m
framework = arduino
upload_speed = 115200
flash_mode = dout
```

### Key Configuration Details

- **Framework**: Arduino
- **Board**: ESP-01 with 1MB flash
- **Upload Speed**: 115200 baud
- **Flash Mode**: DOUT (required for ESP-01S)
- **Build Type**: Release (optimized for production)

### Dependencies

The project requires the following libraries:

```ini
lib_deps =
    esp32async/ESPAsyncWebServer@^3.8.0
    bblanchon/ArduinoJson@^7.4.2
    arkhipenko/TaskScheduler@^3.8.5
    ESP8266HTTPClient@1.2
    ESP8266WiFi@1.0
```

## Building

### Basic Build Commands

Build the firmware for specific chip:

```bash
# Build for ESP-01 1MB
./ctl.sh build target esp01_1m

# Build for D1 Mini
./ctl.sh build target d1_mini

# Build with verbose output
pio run -v
```

### Build Targets

Available build targets:

| Target | Description | Flash Size |
|--------|-------------|------------|
| `esp01_1m` | ESP-01 with 1MB flash | 1MB |
| `d1_mini` | D1 Mini with 4MB flash | 4MB |

### Build Artifacts

After successful build:

- **Firmware**: `.pio/build/{target}/firmware.bin`
- **ELF File**: `.pio/build/{target}/firmware.elf`
- **Map File**: `.pio/build/{target}/firmware.map`

## Flashing

### Direct USB Flashing

Flash firmware to a connected device:

```bash
# Flash ESP-01
./ctl.sh flash target esp01_1m

# Flash D1 Mini
./ctl.sh flash target d1_mini

# Manual flash command
pio run --target upload
```

### Flash Settings

- **Upload Speed**: 115200 baud (optimal for ESP-01)
- **Flash Mode**: DOUT (required for ESP-01S)
- **Reset Method**: Hardware reset or manual reset

### Troubleshooting Flashing

Common flashing issues:

1. **Connection Failed**: Check USB cable and drivers
2. **Wrong Upload Speed**: Try lower speeds (9600, 57600)
3. **Flash Mode Error**: Ensure DOUT mode for ESP-01S
4. **Permission Denied**: Run with sudo or add user to dialout group

## Over-The-Air Updates

### Single Node Update

Update a specific node:

```bash
# Update specific node
./ctl.sh ota update 192.168.1.100 esp01_1m

# Update with custom firmware
./ctl.sh ota update 192.168.1.100 esp01_1m custom_firmware.bin
```

### Cluster-Wide Updates

Update all nodes in the cluster:

```bash
# Update all nodes
./ctl.sh ota all esp01_1m
```

### OTA Process

1. **Firmware Upload**: Send firmware to target node
2. **Verification**: Check firmware integrity
3. **Installation**: Install new firmware
4. **Restart**: Node restarts with new firmware
5. **Verification**: Confirm successful update

### OTA Requirements

- **Flash Space**: Minimum 1MB for OTA updates
- **Network**: Stable WiFi connection
- **Power**: Stable power supply during update
- **Memory**: Sufficient RAM for firmware processing

## Cluster Management

### View Cluster Status

```bash
# View all cluster members
./ctl.sh cluster members

# View specific node details
./ctl.sh cluster members --node 192.168.1.100
```

### Cluster Commands

Available cluster management commands:

| Command | Description |
|---------|-------------|
| `members` | List all cluster members |
| `status` | Show cluster health status |
| `discover` | Force discovery process |
| `health` | Check cluster member health |

### Cluster Monitoring

Monitor cluster health in real-time:

```bash
# Watch cluster status
watch -n 5 './ctl.sh cluster members'

# Monitor specific metrics
./ctl.sh cluster members | jq '.members[] | {hostname, status, latency}'
```

## Development Workflow

### Local Development

1. **Setup Environment**:
   ```bash
   git clone <repository>
   cd spore
   pio run
   ```

2. **Make Changes**:
   - Edit source files in `src/`
   - Modify headers in `include/`
   - Update configuration in `platformio.ini`

3. **Test Changes**:
   ```bash
   pio run
   pio check
   ```

### Testing

Run various tests:

```bash
# Code quality check
pio check

# Unit tests (if available)
pio test

# Memory usage analysis
pio run --target size
```

### Debugging

Enable debug output:

```bash
# Serial monitoring
pio device monitor

# Build with debug symbols
pio run --environment esp01_1m --build-flags -DDEBUG
```

## Configuration Management

### Environment Setup

Create a `.env` file in your project root:

```bash
# API node IP for cluster management
export API_NODE=192.168.1.100
```

### Configuration Files

Key configuration files:

- **`platformio.ini`**: Build and upload configuration
- **`src/Config.cpp`**: Application configuration
- **`.env`**: Environment variables
- **`ctl.sh`**: Build and deployment scripts

### Configuration Options

Available configuration options:

| Option | Default | Description |
|--------|---------|-------------|
| `CLUSTER_PORT` | 4210 | UDP discovery port |
| `DISCOVERY_INTERVAL` | 1000 | Discovery packet interval (ms) |
| `HEALTH_CHECK_INTERVAL` | 1000 | Health check interval (ms) |
| `API_SERVER_PORT` | 80 | HTTP API server port |

## Deployment Strategies

### Development Deployment

For development and testing:

1. **Build**: `pio run`
2. **Flash**: `pio run --target upload`
3. **Monitor**: `pio device monitor`

### Production Deployment

For production systems:

1. **Build Release**: `pio run --environment esp01_1m`
2. **OTA Update**: `./ctl.sh ota update <ip> esp01_1m`
3. **Verify**: Check node status via API

### Continuous Integration

Automated deployment pipeline:

```yaml
# Example GitHub Actions workflow
- name: Build Firmware
  run: pio run --environment esp01_1m

- name: Deploy to Test Cluster
  run: ./ctl.sh ota all esp01_1m --target test

- name: Deploy to Production
  run: ./ctl.sh ota all esp01_1m --target production
```

## Monitoring and Debugging

### Serial Output

Enable serial monitoring:

```bash
# Basic monitoring
pio device monitor

# With specific baud rate
pio device monitor --baud 115200

# Filter specific messages
pio device monitor | grep "Cluster"
```

### API Monitoring

Monitor system via HTTP API:

```bash
# Check system status
curl -s http://192.168.1.100/api/node/status | jq '.'

# Monitor tasks
curl -s http://192.168.1.100/api/tasks/status | jq '.'

# Check cluster health
curl -s http://192.168.1.100/api/cluster/members | jq '.'
```

### Performance Monitoring

Track system performance:

```bash
# Memory usage over time
watch -n 5 'curl -s http://192.168.1.100/api/node/status | jq ".freeHeap"'

# Task execution status
watch -n 10 'curl -s http://192.168.1.100/api/tasks/status | jq ".summary"'
```

## Troubleshooting

### Common Issues

1. **Discovery Failures**: Check UDP port 4210 is not blocked
2. **WiFi Connection**: Verify SSID/password in Config.cpp
3. **OTA Updates**: Ensure sufficient flash space (1MB minimum)
4. **Cluster Split**: Check network connectivity between nodes

### Debug Commands

Useful debugging commands:

```bash
# Check network connectivity
ping 192.168.1.100

# Test UDP port
nc -u 192.168.1.100 4210

# Check HTTP API
curl -v http://192.168.1.100/api/node/status

# Monitor system resources
./ctl.sh cluster members | jq '.members[] | {hostname, status, resources.freeHeap}'
```

### Performance Issues

Common performance problems:

- **Memory Leaks**: Monitor free heap over time
- **Network Congestion**: Check discovery intervals
- **Task Overload**: Review task execution intervals
- **WiFi Interference**: Check channel and signal strength

## Best Practices

### Code Organization

1. **Modular Design**: Keep components loosely coupled
2. **Clear Interfaces**: Define clear APIs between components
3. **Error Handling**: Implement proper error handling and logging
4. **Resource Management**: Efficient memory and resource usage

### Testing Strategy

1. **Unit Tests**: Test individual components
2. **Integration Tests**: Test component interactions
3. **System Tests**: Test complete system functionality
4. **Performance Tests**: Monitor resource usage and performance

### Deployment Strategy

1. **Staged Rollout**: Deploy to test cluster first
2. **Rollback Plan**: Maintain ability to rollback updates
3. **Monitoring**: Monitor system health during deployment
4. **Documentation**: Keep deployment procedures updated

## Related Documentation

- **[Architecture Guide](./Architecture.md)** - System architecture overview
- **[Task Management](./TaskManagement.md)** - Background task system
- **[API Reference](./API.md)** - REST API documentation
- **[OpenAPI Specification](../api/)** - Machine-readable API specification