spore/docs/Development.md

# Development & Deployment Guide

## Prerequisites

### Required Tools

- **PlatformIO Core** or **PlatformIO IDE**
- **ESP8266 development tools**
- **`jq`** for JSON processing in scripts
- **Git** for version control

### System Requirements

- **Operating System**: Linux, macOS, or Windows
- **Python**: 3.7+ (for PlatformIO)
- **Memory**: 4GB+ RAM recommended
- **Storage**: 2GB+ free space for development environment

## Project Structure

```
spore/
├── src/                    # Source code (framework under src/spore)
│   └── spore/
│       ├── Spore.cpp               # Framework lifecycle (setup/begin/loop)
│       ├── core/                   # Core components
│       │   ├── ApiServer.cpp       # HTTP API server implementation
│       │   ├── ClusterManager.cpp  # Cluster management logic
│       │   ├── NetworkManager.cpp  # WiFi and network handling
│       │   ├── TaskManager.cpp     # Background task management
│       │   └── NodeContext.cpp     # Central context and events
│       ├── services/               # Built-in services (implement Service interface)
│       │   ├── NodeService.cpp     # registerEndpoints() + registerTasks()
│       │   ├── NetworkService.cpp  # registerEndpoints() + registerTasks()
│       │   ├── ClusterService.cpp  # registerEndpoints() + registerTasks()
│       │   ├── TaskService.cpp     # registerEndpoints() + registerTasks()
│       │   ├── StaticFileService.cpp # registerEndpoints() + registerTasks()
│       │   └── MonitoringService.cpp # registerEndpoints() + registerTasks()
│       └── types/                  # Shared types
├── include/                # Header files
├── examples/               # Example apps per env (base, relay, neopattern)
├── docs/                   # Documentation
├── api/                    # OpenAPI specification
├── platformio.ini          # PlatformIO configuration
└── ctl.sh                  # Build and deployment scripts
```

## PlatformIO Configuration

### Framework and Board

The project uses PlatformIO with the following configuration (excerpt):

```ini
[platformio]
default_envs = base
src_dir = .
data_dir = ${PROJECT_DIR}/examples/${PIOENV}/data

[common]
monitor_speed = 115200
lib_deps = 
    esp32async/ESPAsyncWebServer@^3.8.0
    bblanchon/ArduinoJson@^7.4.2

[env:base]
platform = platformio/espressif8266@^4.2.1
board = esp01_1m
framework = arduino
upload_speed = 115200
monitor_speed = 115200
board_build.f_cpu = 80000000L
board_build.flash_mode = qio
board_build.filesystem = littlefs
; note: somehow partition table is not working, so we need to use the ldscript
board_build.ldscript = eagle.flash.1m64.ld
lib_deps = ${common.lib_deps}
build_src_filter = 
    +<examples/base/*.cpp>
    +<src/spore/*.cpp>
    +<src/spore/core/*.cpp>
    +<src/spore/services/*.cpp>
    +<src/spore/types/*.cpp>
    +<src/spore/util/*.cpp>
    +<src/internal/*.cpp>

[env:d1_mini]
platform = platformio/espressif8266@^4.2.1
board = d1_mini
framework = arduino
upload_speed = 115200
monitor_speed = 115200
board_build.filesystem = littlefs
board_build.flash_mode = dio           ; D1 Mini uses DIO on 4 Mbit flash
board_build.flash_size = 4M
board_build.ldscript = eagle.flash.4m1m.ld
lib_deps = ${common.lib_deps}
build_src_filter = 
    +<examples/base/*.cpp>
    +<src/spore/*.cpp>
    +<src/spore/core/*.cpp>
    +<src/spore/services/*.cpp>
    +<src/spore/types/*.cpp>
    +<src/spore/util/*.cpp>
    +<src/internal/*.cpp>
```

### Dependencies

The project requires the following libraries (resolved via PlatformIO):

```ini
lib_deps =
    esp32async/ESPAsyncWebServer@^3.8.0
    bblanchon/ArduinoJson@^7.4.2
```

### Filesystem, Linker Scripts, and Flash Layout

This project uses LittleFS as the filesystem on ESP8266. Flash layout is controlled by the linker script (ldscript) selected per environment in `platformio.ini`.

- **Filesystem**: LittleFS (replaces SPIFFS). Although ldscripts reference SPIFFS in their names, the reserved region is used by LittleFS transparently.
- **Why ldscript matters**: It defines maximum sketch size and the size of the filesystem area in flash.

| Board / Env | Flash size | ldscript | Filesystem | FS size | Notes |
|-------------|------------|---------|------------|---------|-------|
| esp01_1m    | 1MB        | `eagle.flash.1m64.ld` | LittleFS | 64KB | Prioritizes sketch size on 1MB modules. OTA is constrained on 1MB.
| d1_mini     | 4MB        | `eagle.flash.4m1m.ld` | LittleFS | 1MB  | Standard 4M/1M layout; ample space for firmware and data.

Configured in `platformio.ini`:

```ini
[env:esp01_1m]
board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.1m64.ld

[env:d1_mini]
board_build.filesystem = littlefs
board_build.ldscript = eagle.flash.4m1m.ld
```

Notes:
- The ldscript name indicates total flash and filesystem size (e.g., `4m1m` = 4MB flash with 1MB FS; `1m64` = 1MB flash with 64KB FS).
- LittleFS works within the filesystem region defined by the ldscript, despite SPIFFS naming.
- If you need a different FS size, select an appropriate ldscript variant and keep `board_build.filesystem = littlefs`.
- On ESP8266, custom partition CSVs are not used for layout; the linker script defines the flash map. This project removed prior `board_build.partitions` usage in favor of explicit `board_build.ldscript` entries per environment.

## Building

### Basic Build Commands

Build the firmware for specific chip:

```bash
# Build for ESP-01 1MB
./ctl.sh build target esp01_1m

# Build for D1 Mini
./ctl.sh build target d1_mini

# Build with verbose output
pio run -v
```

### Build Targets

Available build targets:

| Target | Description | Flash Size |
|--------|-------------|------------|
| `esp01_1m` | ESP-01 with 1MB flash | 1MB |
| `d1_mini` | D1 Mini with 4MB flash | 4MB |

### Build Artifacts

After successful build:

- **Firmware**: `.pio/build/{target}/firmware.bin`
- **ELF File**: `.pio/build/{target}/firmware.elf`
- **Map File**: `.pio/build/{target}/firmware.map`

## Flashing

### Direct USB Flashing

Flash firmware to a connected device:

```bash
# Flash ESP-01
./ctl.sh flash target esp01_1m

# Flash D1 Mini
./ctl.sh flash target d1_mini

# Manual flash command
pio run --target upload
```

### Flash Settings

- **Upload Speed**: 115200 baud (optimal for ESP-01)
- **Flash Mode**: DOUT (required for ESP-01S)
- **Reset Method**: Hardware reset or manual reset

### Troubleshooting Flashing

Common flashing issues:

1. **Connection Failed**: Check USB cable and drivers
2. **Wrong Upload Speed**: Try lower speeds (9600, 57600)
3. **Flash Mode Error**: Ensure DOUT mode for ESP-01S
4. **Permission Denied**: Run with sudo or add user to dialout group

## Over-The-Air Updates

### Single Node Update

Update a specific node:

```bash
# Update specific node
./ctl.sh ota update 192.168.1.100 esp01_1m

# Update with custom firmware
./ctl.sh ota update 192.168.1.100 esp01_1m custom_firmware.bin
```

### Cluster-Wide Updates

Update all nodes in the cluster:

```bash
# Update all nodes
./ctl.sh ota all esp01_1m
```

### OTA Process

1. **Firmware Upload**: Send firmware to target node
2. **Verification**: Check firmware integrity
3. **Installation**: Install new firmware
4. **Restart**: Node restarts with new firmware
5. **Verification**: Confirm successful update

### OTA Requirements

- **Flash Space**: Minimum 1MB for OTA updates
- **Network**: Stable WiFi connection
- **Power**: Stable power supply during update
- **Memory**: Sufficient RAM for firmware processing

## Cluster Management

### View Cluster Status

```bash
# View all cluster members
./ctl.sh cluster members

# View specific node details
./ctl.sh cluster members --node 192.168.1.100
```

### Cluster Commands

Available cluster management commands:

| Command | Description |
|---------|-------------|
| `members` | List all cluster members |
| `status` | Show cluster health status |
| `discover` | Force discovery process |
| `health` | Check cluster member health |

### Cluster Monitoring

Monitor cluster health in real-time:

```bash
# Watch cluster status
watch -n 5 './ctl.sh cluster members'

# Monitor specific metrics
./ctl.sh cluster members | jq '.members[] | {hostname, status, latency}'
```

## Development Workflow

### Local Development

1. **Setup Environment**:
   ```bash
   git clone <repository>
   cd spore
   pio run
   ```

2. **Make Changes**:
   - Edit source files in `src/`
   - Modify headers in `include/`
   - Update configuration in `platformio.ini`

3. **Test Changes**:
   ```bash
   pio run
   pio check
   ```

### Testing

Run various tests:

```bash
# Code quality check
pio check

# Unit tests (if available)
pio test

# Memory usage analysis
pio run --target size
```

### Debugging

Enable debug output:

```bash
# Serial monitoring
pio device monitor

# Build with debug symbols
pio run --environment esp01_1m --build-flags -DDEBUG
```

## Configuration Management

### Environment Setup

Create a `.env` file in your project root:

```bash
# API node IP for cluster management
export API_NODE=192.168.1.100
```

### Configuration Files

Key configuration files:

- **`platformio.ini`**: Build and upload configuration
- **`src/spore/types/Config.cpp`**: Default runtime configuration
- **`.env`**: Environment variables
- **`ctl.sh`**: Build and deployment scripts

### Configuration Options

Available configuration options:

| Option | Default | Description |
|--------|---------|-------------|
| `CLUSTER_PORT` | 4210 | UDP discovery port |
| `DISCOVERY_INTERVAL` | 1000 | Discovery packet interval (ms) |
| `HEALTH_CHECK_INTERVAL` | 1000 | Health check interval (ms) |
| `API_SERVER_PORT` | 80 | HTTP API server port |

## Deployment Strategies

### Development Deployment

For development and testing:

1. **Build**: `pio run`
2. **Flash**: `pio run --target upload`
3. **Monitor**: `pio device monitor`

### Production Deployment

For production systems:

1. **Build Release**: `pio run --environment esp01_1m`
2. **OTA Update**: `./ctl.sh ota update <ip> esp01_1m`
3. **Verify**: Check node status via API

### Continuous Integration

Automated deployment pipeline:

```yaml
# Example GitHub Actions workflow
- name: Build Firmware
  run: pio run --environment esp01_1m

- name: Deploy to Test Cluster
  run: ./ctl.sh ota all esp01_1m --target test

- name: Deploy to Production
  run: ./ctl.sh ota all esp01_1m --target production
```

## Monitoring and Debugging

### Serial Output

Enable serial monitoring:

```bash
# Basic monitoring
pio device monitor

# With specific baud rate
pio device monitor --baud 115200

# Filter specific messages
pio device monitor | grep "Cluster"
```

### API Monitoring

Monitor system via HTTP API:

```bash
# Check system status
curl -s http://192.168.1.100/api/node/status | jq '.'

# Monitor tasks
curl -s http://192.168.1.100/api/tasks/status | jq '.'

# Check cluster health
curl -s http://192.168.1.100/api/cluster/members | jq '.'
```

### Performance Monitoring

Track system performance:

```bash
# Memory usage over time
watch -n 5 'curl -s http://192.168.1.100/api/node/status | jq ".freeHeap"'

# Task execution status
watch -n 10 'curl -s http://192.168.1.100/api/tasks/status | jq ".summary"'
```

## Troubleshooting

### Common Issues

1. **Discovery Failures**: Check UDP port 4210 is not blocked
2. **WiFi Connection**: Verify SSID/password in Config.cpp
3. **OTA Updates**: Ensure sufficient flash space (1MB minimum)
4. **Cluster Split**: Check network connectivity between nodes

### Debug Commands

Useful debugging commands:

```bash
# Check network connectivity
ping 192.168.1.100

# Test UDP port
nc -u 192.168.1.100 4210

# Check HTTP API
curl -v http://192.168.1.100/api/node/status

# Monitor system resources
./ctl.sh cluster members | jq '.members[] | {hostname, status, resources.freeHeap}'
```

### Performance Issues

Common performance problems:

- **Memory Leaks**: Monitor free heap over time
- **Network Congestion**: Check discovery intervals
- **Task Overload**: Review task execution intervals
- **WiFi Interference**: Check channel and signal strength

## Best Practices

### Code Organization

1. **Modular Design**: Keep components loosely coupled
2. **Clear Interfaces**: Define clear APIs between components
3. **Error Handling**: Implement proper error handling and logging
4. **Resource Management**: Efficient memory and resource usage

### Testing Strategy

1. **Unit Tests**: Test individual components
2. **Integration Tests**: Test component interactions
3. **System Tests**: Test complete system functionality
4. **Performance Tests**: Monitor resource usage and performance

### Deployment Strategy

1. **Staged Rollout**: Deploy to test cluster first
2. **Rollback Plan**: Maintain ability to rollback updates
3. **Monitoring**: Monitor system health during deployment
4. **Documentation**: Keep deployment procedures updated

## Related Documentation

- **[Architecture Guide](./Architecture.md)** - System architecture overview
- **[Task Management](./TaskManagement.md)** - Background task system
- **[API Reference](./API.md)** - REST API documentation
- **[OpenAPI Specification](../api/)** - Machine-readable API specification