# Development & Deployment Guide ## Prerequisites ### Required Tools - **PlatformIO Core** or **PlatformIO IDE** - **ESP8266 development tools** - **`jq`** for JSON processing in scripts - **Git** for version control ### System Requirements - **Operating System**: Linux, macOS, or Windows - **Python**: 3.7+ (for PlatformIO) - **Memory**: 4GB+ RAM recommended - **Storage**: 2GB+ free space for development environment ## Project Structure ``` spore/ ├── src/ # Source code (framework under src/spore) │ └── spore/ │ ├── Spore.cpp # Framework lifecycle (setup/begin/loop) │ ├── core/ # Core components │ │ ├── ApiServer.cpp # HTTP API server implementation │ │ ├── ClusterManager.cpp # Cluster management logic │ │ ├── NetworkManager.cpp # WiFi and network handling │ │ ├── TaskManager.cpp # Background task management │ │ └── NodeContext.cpp # Central context and events │ ├── services/ # Built-in services (implement Service interface) │ │ ├── NodeService.cpp # registerEndpoints() + registerTasks() │ │ ├── NetworkService.cpp # registerEndpoints() + registerTasks() │ │ ├── ClusterService.cpp # registerEndpoints() + registerTasks() │ │ ├── TaskService.cpp # registerEndpoints() + registerTasks() │ │ ├── StaticFileService.cpp # registerEndpoints() + registerTasks() │ │ └── MonitoringService.cpp # registerEndpoints() + registerTasks() │ └── types/ # Shared types ├── include/ # Header files ├── examples/ # Example apps per env (base, relay, neopattern) ├── docs/ # Documentation ├── api/ # OpenAPI specification ├── platformio.ini # PlatformIO configuration └── ctl.sh # Build and deployment scripts ``` ## PlatformIO Configuration ### Framework and Board The project uses PlatformIO with the following configuration (excerpt): ```ini [platformio] default_envs = base src_dir = . data_dir = ${PROJECT_DIR}/examples/${PIOENV}/data [common] monitor_speed = 115200 lib_deps = esp32async/ESPAsyncWebServer@^3.8.0 bblanchon/ArduinoJson@^7.4.2 [env:base] platform = platformio/espressif8266@^4.2.1 board = esp01_1m framework = arduino upload_speed = 115200 monitor_speed = 115200 board_build.f_cpu = 80000000L board_build.flash_mode = qio board_build.filesystem = littlefs ; note: somehow partition table is not working, so we need to use the ldscript board_build.ldscript = eagle.flash.1m64.ld lib_deps = ${common.lib_deps} build_src_filter = + + + + + + + [env:d1_mini] platform = platformio/espressif8266@^4.2.1 board = d1_mini framework = arduino upload_speed = 115200 monitor_speed = 115200 board_build.filesystem = littlefs board_build.flash_mode = dio ; D1 Mini uses DIO on 4 Mbit flash board_build.flash_size = 4M board_build.ldscript = eagle.flash.4m1m.ld lib_deps = ${common.lib_deps} build_src_filter = + + + + + + + ``` ### Dependencies The project requires the following libraries (resolved via PlatformIO): ```ini lib_deps = esp32async/ESPAsyncWebServer@^3.8.0 bblanchon/ArduinoJson@^7.4.2 ``` ### Filesystem, Linker Scripts, and Flash Layout This project uses LittleFS as the filesystem on ESP8266. Flash layout is controlled by the linker script (ldscript) selected per environment in `platformio.ini`. - **Filesystem**: LittleFS (replaces SPIFFS). Although ldscripts reference SPIFFS in their names, the reserved region is used by LittleFS transparently. - **Why ldscript matters**: It defines maximum sketch size and the size of the filesystem area in flash. | Board / Env | Flash size | ldscript | Filesystem | FS size | Notes | |-------------|------------|---------|------------|---------|-------| | esp01_1m | 1MB | `eagle.flash.1m64.ld` | LittleFS | 64KB | Prioritizes sketch size on 1MB modules. OTA is constrained on 1MB. | d1_mini | 4MB | `eagle.flash.4m1m.ld` | LittleFS | 1MB | Standard 4M/1M layout; ample space for firmware and data. Configured in `platformio.ini`: ```ini [env:esp01_1m] board_build.filesystem = littlefs board_build.ldscript = eagle.flash.1m64.ld [env:d1_mini] board_build.filesystem = littlefs board_build.ldscript = eagle.flash.4m1m.ld ``` Notes: - The ldscript name indicates total flash and filesystem size (e.g., `4m1m` = 4MB flash with 1MB FS; `1m64` = 1MB flash with 64KB FS). - LittleFS works within the filesystem region defined by the ldscript, despite SPIFFS naming. - If you need a different FS size, select an appropriate ldscript variant and keep `board_build.filesystem = littlefs`. - On ESP8266, custom partition CSVs are not used for layout; the linker script defines the flash map. This project removed prior `board_build.partitions` usage in favor of explicit `board_build.ldscript` entries per environment. ## Building ### Basic Build Commands Build the firmware for specific chip: ```bash # Build for ESP-01 1MB ./ctl.sh build target esp01_1m # Build for D1 Mini ./ctl.sh build target d1_mini # Build with verbose output pio run -v ``` ### Build Targets Available build targets: | Target | Description | Flash Size | |--------|-------------|------------| | `esp01_1m` | ESP-01 with 1MB flash | 1MB | | `d1_mini` | D1 Mini with 4MB flash | 4MB | ### Build Artifacts After successful build: - **Firmware**: `.pio/build/{target}/firmware.bin` - **ELF File**: `.pio/build/{target}/firmware.elf` - **Map File**: `.pio/build/{target}/firmware.map` ## Flashing ### Direct USB Flashing Flash firmware to a connected device: ```bash # Flash ESP-01 ./ctl.sh flash target esp01_1m # Flash D1 Mini ./ctl.sh flash target d1_mini # Manual flash command pio run --target upload ``` ### Flash Settings - **Upload Speed**: 115200 baud (optimal for ESP-01) - **Flash Mode**: DOUT (required for ESP-01S) - **Reset Method**: Hardware reset or manual reset ### Troubleshooting Flashing Common flashing issues: 1. **Connection Failed**: Check USB cable and drivers 2. **Wrong Upload Speed**: Try lower speeds (9600, 57600) 3. **Flash Mode Error**: Ensure DOUT mode for ESP-01S 4. **Permission Denied**: Run with sudo or add user to dialout group ## Over-The-Air Updates ### Single Node Update Update a specific node: ```bash # Update specific node ./ctl.sh ota update 192.168.1.100 esp01_1m # Update with custom firmware ./ctl.sh ota update 192.168.1.100 esp01_1m custom_firmware.bin ``` ### Cluster-Wide Updates Update all nodes in the cluster: ```bash # Update all nodes ./ctl.sh ota all esp01_1m ``` ### OTA Process 1. **Firmware Upload**: Send firmware to target node 2. **Verification**: Check firmware integrity 3. **Installation**: Install new firmware 4. **Restart**: Node restarts with new firmware 5. **Verification**: Confirm successful update ### OTA Requirements - **Flash Space**: Minimum 1MB for OTA updates - **Network**: Stable WiFi connection - **Power**: Stable power supply during update - **Memory**: Sufficient RAM for firmware processing ## Cluster Management ### View Cluster Status ```bash # View all cluster members ./ctl.sh cluster members # View specific node details ./ctl.sh cluster members --node 192.168.1.100 ``` ### Cluster Commands Available cluster management commands: | Command | Description | |---------|-------------| | `members` | List all cluster members | | `status` | Show cluster health status | | `discover` | Force discovery process | | `health` | Check cluster member health | ### Cluster Monitoring Monitor cluster health in real-time: ```bash # Watch cluster status watch -n 5 './ctl.sh cluster members' # Monitor specific metrics ./ctl.sh cluster members | jq '.members[] | {hostname, status, latency}' ``` ## Development Workflow ### Local Development 1. **Setup Environment**: ```bash git clone cd spore pio run ``` 2. **Make Changes**: - Edit source files in `src/` - Modify headers in `include/` - Update configuration in `platformio.ini` 3. **Test Changes**: ```bash pio run pio check ``` ### Testing Run various tests: ```bash # Code quality check pio check # Unit tests (if available) pio test # Memory usage analysis pio run --target size ``` ### Debugging Enable debug output: ```bash # Serial monitoring pio device monitor # Build with debug symbols pio run --environment esp01_1m --build-flags -DDEBUG ``` ## Configuration Management ### Environment Setup Create a `.env` file in your project root: ```bash # API node IP for cluster management export API_NODE=192.168.1.100 ``` ### Configuration Files Key configuration files: - **`platformio.ini`**: Build and upload configuration - **`src/spore/types/Config.cpp`**: Default runtime configuration - **`.env`**: Environment variables - **`ctl.sh`**: Build and deployment scripts ### Configuration Options Available configuration options: | Option | Default | Description | |--------|---------|-------------| | `CLUSTER_PORT` | 4210 | UDP discovery port | | `DISCOVERY_INTERVAL` | 1000 | Discovery packet interval (ms) | | `HEALTH_CHECK_INTERVAL` | 1000 | Health check interval (ms) | | `API_SERVER_PORT` | 80 | HTTP API server port | ## Deployment Strategies ### Development Deployment For development and testing: 1. **Build**: `pio run` 2. **Flash**: `pio run --target upload` 3. **Monitor**: `pio device monitor` ### Production Deployment For production systems: 1. **Build Release**: `pio run --environment esp01_1m` 2. **OTA Update**: `./ctl.sh ota update esp01_1m` 3. **Verify**: Check node status via API ### Continuous Integration Automated deployment pipeline: ```yaml # Example GitHub Actions workflow - name: Build Firmware run: pio run --environment esp01_1m - name: Deploy to Test Cluster run: ./ctl.sh ota all esp01_1m --target test - name: Deploy to Production run: ./ctl.sh ota all esp01_1m --target production ``` ## Monitoring and Debugging ### Serial Output Enable serial monitoring: ```bash # Basic monitoring pio device monitor # With specific baud rate pio device monitor --baud 115200 # Filter specific messages pio device monitor | grep "Cluster" ``` ### API Monitoring Monitor system via HTTP API: ```bash # Check system status curl -s http://192.168.1.100/api/node/status | jq '.' # Monitor tasks curl -s http://192.168.1.100/api/tasks/status | jq '.' # Check cluster health curl -s http://192.168.1.100/api/cluster/members | jq '.' ``` ### Performance Monitoring Track system performance: ```bash # Memory usage over time watch -n 5 'curl -s http://192.168.1.100/api/node/status | jq ".freeHeap"' # Task execution status watch -n 10 'curl -s http://192.168.1.100/api/tasks/status | jq ".summary"' ``` ## Troubleshooting ### Common Issues 1. **Discovery Failures**: Check UDP port 4210 is not blocked 2. **WiFi Connection**: Verify SSID/password in Config.cpp 3. **OTA Updates**: Ensure sufficient flash space (1MB minimum) 4. **Cluster Split**: Check network connectivity between nodes ### Debug Commands Useful debugging commands: ```bash # Check network connectivity ping 192.168.1.100 # Test UDP port nc -u 192.168.1.100 4210 # Check HTTP API curl -v http://192.168.1.100/api/node/status # Monitor system resources ./ctl.sh cluster members | jq '.members[] | {hostname, status, resources.freeHeap}' ``` ### Performance Issues Common performance problems: - **Memory Leaks**: Monitor free heap over time - **Network Congestion**: Check discovery intervals - **Task Overload**: Review task execution intervals - **WiFi Interference**: Check channel and signal strength ## Best Practices ### Code Organization 1. **Modular Design**: Keep components loosely coupled 2. **Clear Interfaces**: Define clear APIs between components 3. **Error Handling**: Implement proper error handling and logging 4. **Resource Management**: Efficient memory and resource usage ### Testing Strategy 1. **Unit Tests**: Test individual components 2. **Integration Tests**: Test component interactions 3. **System Tests**: Test complete system functionality 4. **Performance Tests**: Monitor resource usage and performance ### Deployment Strategy 1. **Staged Rollout**: Deploy to test cluster first 2. **Rollback Plan**: Maintain ability to rollback updates 3. **Monitoring**: Monitor system health during deployment 4. **Documentation**: Keep deployment procedures updated ## Related Documentation - **[Architecture Guide](./Architecture.md)** - System architecture overview - **[Task Management](./TaskManagement.md)** - Background task system - **[API Reference](./API.md)** - REST API documentation - **[OpenAPI Specification](../api/)** - Machine-readable API specification