# SPORE - Sprocket Orchestration Engine

A totally based cluster engine for ESP8266 microcontrollers with automatic node discovery, health monitoring, and over-the-air updates. Built for real-time control of distributed embedded systems.

## Core

Cluster engine firmware for ESP8266 microcontrollers

Repository: spore

### Auto Discovery

UDP-based node discovery with automatic cluster membership

### Health Monitoring

Real-time node status tracking with resource monitoring and automatic failover

### Event System

Local and cluster-wide event publishing/subscription with WebSocket and UDP streaming API

### OTA Updates

Seamless firmware updates across the cluster via HTTP API

### Service Registry

Dynamic API endpoint discovery and registration for custom services

### Task Scheduler

Cooperative multitasking system for background operations

### Quick Start

    #include <Arduino.h>
    #include "spore/Spore.h"
    #include "spore/Service.h"
    #include "spore/core/ApiServer.h"
    #include "spore/core/TaskManager.h"

    class HelloService : public Service {
    public:
        HelloService() {}

        const char* getName() const override { return "HelloService"; }

        void registerEndpoints(ApiServer& api) override {
            api.registerEndpoint("/api/hello", HTTP_GET, [](AsyncWebServerRequest* req) {
                req->send(200, "application/json", "{\"message\":\"hello\"}");
            });
        }

        void registerTasks(TaskManager& taskManager) override {
            taskManager.registerTask("heartbeat", 1000, [this]() { this->heartbeat(); });
        }

    private:
        void heartbeat() {
            // e.g., blink LED, publish telemetry, etc.
        }
    };

    Spore spore({
        {"app", "my_app"},
        {"role", "controller"}
    });

    void setup() {
        spore.setup();
        spore.registerService(new HelloService());
        spore.begin();
    }

    void loop() {
        spore.loop();
    }

### Technical Specifications

- Supported Hardware: ESP-01/ESP-01S (1MB Flash), Wemos D1 (4MB Flash)
- Discovery Protocol: Discovery and heartbeat via UDP broadcast
- API Interface: RESTful HTTP + WebSocket and UDP streaming
- Dependencies: ESPAsyncWebServer, ArduinoJson
- Framework: Arduino with PlatformIO build system

## Gateway

Gateway service providing UDP-based node discovery, cluster management, and API endpoints

Repository: spore-gateway

### Node Discovery

UDP-based discovery listens for heartbeat messages from SPORE nodes on port 4210 with automatic cluster membership

### Cluster Management

Real-time node status tracking, primary node selection, and automatic failover for high availability

### WebSocket Server

Real-time cluster updates and event broadcasting to all connected WebSocket clients

### MQTT Integration

Subscribe to all MQTT topics and forward messages to WebSocket clients for unified event streaming

### Rollout System

Orchestrated firmware updates across multiple nodes with parallel processing and progress tracking

### Proxy API

Generic proxy calls to SPORE node capabilities with automatic failover and load balancing

### Architecture & Integration

SPORE Gateway acts as the central orchestrator for the SPORE cluster, bridging the gap between SPORE nodes and the UI. It integrates seamlessly with the SPORE ecosystem and external systems through multiple protocols.

### Core Components

- UDP Discovery: Listens on port 4210 for node heartbeats and maintains cluster membership in real-time
- HTTP API Server: RESTful endpoints for cluster management, node control, and firmware operations
- WebSocket Server: Broadcasts cluster updates, events, and rollout progress to all connected clients
- MQTT Client: Optional integration to subscribe to MQTT brokers and forward messages via WebSocket
- Registry Proxy: Integrates with spore-registry for firmware storage and retrieval
- Failover Logic: Automatic primary node switching when nodes go offline

### MQTT Integration

Enable MQTT integration to stream events from external IoT devices and systems:

    # Start gateway with MQTT integration
    ./spore-gateway -mqtt tcp://localhost:1883

    # With authentication
    MQTT_USER=username MQTT_PASSWORD=password \
      ./spore-gateway -mqtt tcp://broker.example.com:1883

When enabled, the gateway subscribes to all MQTT topics (#) and forwards messages to connected WebSocket clients, allowing the UI to display both SPORE cluster events and external MQTT events in real-time.

### Rollout Orchestration

The rollout system manages parallel firmware updates across multiple SPORE nodes:

- Automatic Node Discovery: Identifies matching nodes based on firmware labels and current versions
- Parallel Processing: Updates multiple nodes simultaneously using goroutines
- Real-time Progress: WebSocket broadcasts show update status for each node
- Error Recovery: Continues rollout even if individual nodes fail
- Registry Integration: Seamless integration with spore-registry for firmware storage

### Technology Stack

- Language: Go (Golang)
- Framework: Standard library net/http with Go 1.22 ServeMux
- WebSocket: gorilla/websocket for real-time communication
- MQTT: Eclipse Paho MQTT client library
- Logging: Structured logging with logrus
- Concurrency: Goroutines and channels for parallel operations

## Registry

Storage backend and update server for managing versioned firmware binaries

Repository: spore-registry

### Hierarchical Storage

Firmware binaries stored in hierarchical structure: registry/{name}/{version}/firmware.bin with semantic versioning support

### SQLite Metadata

Metadata storage with labels for flexible querying, version management, and deployment tracking

### REST API

Simple HTTP endpoints for upload, listing, download, and update operations with full CRUD support

### Rollout Integration

Seamlessly integrated with Gateway for orchestrated firmware deployment across cluster nodes

### Version Management

Semantic versioning support with unique name-version constraints and version history tracking

### Label System

Flexible key-value labels for organizing and querying firmware by app, role, or custom attributes

### Architecture & Role

SPORE Registry serves as the central firmware storage and management system for the SPORE ecosystem. It provides versioned storage for firmware binaries, enabling the Gateway to orchestrate firmware rollouts across cluster nodes. The registry integrates seamlessly with the Gateway's rollout system, providing the firmware binaries needed for over-the-air updates.

### Storage Structure

Firmware is stored hierarchically in the file system:

    registry/
    ├── base/
    │   ├── 1.0.0/
    │   │   └── firmware.bin
    │   └── 1.1.0/
    │       └── firmware.bin
    └── sensor/
        └── 2.0.0/
            └── firmware.bin

This structure allows easy organization and access to firmware versions by name and version number.

### Integration with Gateway

The registry is tightly integrated with the Gateway's rollout system:

- Firmware Lookup: Gateway queries the registry to find matching firmware for deployment
- Label Matching: Firmware can be organized with labels (app, role, target) for automated deployment
- Version Tracking: Registry maintains version history for rollback and audit purposes
- Binary Distribution: Gateway proxies firmware downloads to nodes during rollout
- Deployment Automation: Full integration with Gateway's rollout orchestration for cluster-wide updates

### Usage Examples

    # Upload firmware
    curl -X POST http://localhost:3002/firmware \
      -F "metadata={\"name\":\"base\",\"version\":\"1.0.0\",\"labels\":{\"app\":\"base\"}}" \
      -F "firmware=@firmware.bin"

    # List firmware
    curl http://localhost:3002/firmware

    # Download firmware
    curl http://localhost:3002/firmware/base/1.0.0 -o firmware.bin

    # Health check
    curl http://localhost:3002/health

### Environment Configuration

The registry supports the following configuration via environment variables:

- PORT: Server port (default: 3002)
- DB_PATH: Database file path (default: ./registry.db)
- REGISTRY_PATH: Firmware storage directory (default: registry)
- MAX_UPLOAD_SIZE: Maximum upload size in bytes (default: 32MB)

### Technology Stack

- Language: Go (Golang)
- Database: SQLite for metadata storage
- Storage: File system for firmware binaries
- API: HTTP REST API with standard library net/http

## UI

Zero-configuration web interface for cluster monitoring and management

Repository: spore-ui

Real-time cluster member overview with auto-discovery

Real-time event flow visualization with interactive graph

Network topology visualization with node relationships

Detailed system metrics and task monitoring

Clusterwide over-the-air firmware updates

### Features

- Cluster monitoring with real-time status updates
- Events visualization showing real-time event flow as an interactive force-directed graph
- Event tracking with animated golden dots showing data flow through cluster topics
- Node details including running tasks and available endpoints
- Direct HTTP API access to all nodes in the cluster
- Over-the-air firmware updates for entire cluster
- WebSocket terminal for direct node interaction
- UDP auto-discovery eliminates hardcoded IP addresses
- Responsive design for all devices

### Technology Stack

- Backend: Express.js, Node.js
- Frontend: Vanilla JavaScript, CSS3, HTML5
- Architecture: Custom component-based framework

## Apps

Application suite built on SPORE

Explore applications that extend SPORE.

- LEDLab — Real-time LED matrix animation streaming and visual preset editor

## LEDLab

Real-time LED matrix animation streaming and visual preset editor

Repository: spore-ledlab

Multi-node management with live canvas preview

Visual preset editor with building blocks

### Firmware Requirements

LEDLab requires SPORE nodes running the PixelStreamController firmware, which handles UDP-based RGB data streaming to NeoPixel strips and matrices. The firmware subscribes to `udp/raw` cluster events and converts incoming hex-encoded pixel data into real-time LED animations.

Key firmware features include support for both strip and matrix configurations, serpentine (zig-zag) wiring patterns, and configurable pixel counts, brightness, and matrix dimensions. The controller automatically remaps pixel coordinates for proper matrix display.

See PixelStream documentation for more information.

### Capabilities

- Multi-node management with individual control
- Real-time canvas preview for each node
- 10+ built-in animation presets (rainbow, lava lamp, aurora, etc.)
- Visual preset editor with reusable building blocks
- Live parameter control with instant feedback
- Import/export custom presets as JSON
- Auto-discovery of SPORE nodes on network
- Configurable FPS (1-60) and matrix dimensions
- Requires SPORE nodes running PixelStreamController

### Building Blocks

Shapes: Circle, Rectangle, Triangle, Blob, Point, Line
Patterns: Trail, Radial, Spiral
Colors: Solid, Gradient, Palette, Rainbow
Animations: Move, Rotate, Pulse, Oscillate, Fade

## Deployment

Complete deployment configurations for running the SPORE stack

Repository: spore-deployment

### Docker Compose

Simple deployment with docker-compose for development and production with one command: `docker compose up -d`

### Nomad Orchestration

Production-ready deployment with HashiCorp Nomad for multi-node clustering, rolling updates, and advanced resource management

### Complete Stack

All SPORE services included: mosquitto MQTT broker, gateway, registry, UI, and LEDLab with proper networking and volumes

### MQTT Integration

Built-in Mosquitto broker for message routing and event streaming with WebSocket support

### Data Persistence

All data persisted in local directories for registry data, MQTT data, and firmware storage

### Health Monitoring

Built-in service discovery, health checks, and real-time status monitoring for all services

### Quick Start

    # Clone the deployment repository
    git clone https://git.dcentral.systems/iot/spore-deployment.git
    cd spore-deployment

    # Start all services with Docker Compose
    docker compose up -d

    # Or start with Nomad (production)
    make nomad-start
    make nomad-job-run

### Services

- mosquitto - MQTT broker (port 1883, WebSocket 9001)
- spore-gateway - Node discovery and WebSocket gateway (port 3001, UDP 4210)
- spore-registry - Firmware registry service (port 8090)
- spore-ledlab - LED animation studio (port 8080)
- spore-ui - Web UI for cluster management (port 3000)

### Access Points

Once deployed, access the services at:

- Web UI: http://localhost:3000
- LED Lab: http://localhost:8080
- Registry API: http://localhost:8090
- Gateway API: http://localhost:3001
- MQTT Broker: tcp://localhost:1883
- Nomad UI: http://localhost:4646 (Nomad deployments only)

## Documentation

### SPORE Core

- Architecture Guide - System design and implementation
- API Reference - Complete REST API documentation
- Development Guide - Build, deployment, configuration
- Task Management - Background task system
- Cluster Broadcast - Event distribution protocol
- Streaming API - WebSocket integration

### SPORE UI

- Getting Started - Installation and setup
- Events View - Real-time event visualization
- UDP Auto Discovery - Network discovery protocol
- API Spec - Backend API reference (OpenAPI)
- Component Framework - Custom UI architecture

### SPORE Gateway

- Getting Started - Installation and usage
- MQTT Integration - MQTT message forwarding
- Rollout Process - Firmware rollout orchestration
- API Spec - Gateway API reference (OpenAPI)

### SPORE Registry

- Getting Started - Installation and usage
- Architecture - System design and structure
- API Spec - Registry API reference (OpenAPI)
- UI Integration - Registry integration with UI

### SPORE LEDLab

- Quick Start - Installation and usage
- Preset Editor - Visual animation builder
- Custom Presets - JSON format and examples
- Building Blocks - Reusable components
- Multi-Node Management - Cluster control
- PixelStream Controller - Required SPORE node firmware

### SPORE Deployment

- Getting Started - Complete deployment guide
- Nomad Deployment - Production orchestration
- Docker Compose - Development configuration

### Getting Started

#### SPORE Core

    # Build and flash firmware
    ./ctl.sh build target esp01
    ./ctl.sh flash target esp01

#### SPORE Gateway

    # Build and start
    go mod tidy
    go build
    ./spore-gateway

    # With MQTT integration
    ./spore-gateway -mqtt tcp://localhost:1883

#### SPORE Registry

    # Build and start
    go mod download
    go run main.go

    # Or build and run
    go build -o spore-registry
    ./spore-registry

#### SPORE UI

    # Install and start
    npm install
    npm start

    # Open browser
    http://localhost:3001

#### SPORE LEDLab

    # Install and start
    npm install
    npm start

    # Open browser
    http://localhost:3000

#### SPORE Deployment

    # Docker Compose (dev)
    docker compose up -d

    # Nomad (production)
    make nomad-start
    make nomad-job-run