iot/spore-gateway
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: MQTT integration

Browse Source
This commit is contained in:
0x1d
2025-10-26 18:37:07 +01:00
parent 55c3aebb3f
commit 42ed391120
13 changed files with 998 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
docs/MQTT.md Normal file
View File

@@ -0,0 +1,370 @@
# MQTT Integration
The SPORE Gateway includes optional MQTT integration that allows subscribing to MQTT brokers and forwarding messages to connected WebSocket clients. This enables integration with IoT devices, sensor networks, and other MQTT-based systems.
## Overview
When enabled, the gateway acts as an MQTT subscriber that:
- Connects to an MQTT broker
- Subscribes to all topics (`#`)
- Forwards received messages to WebSocket clients in real-time
This allows the SPORE UI to display MQTT events alongside SPORE cluster events.
## Features
- **Universal Topic Subscription**: Subscribes to `#` (all topics) to capture all messages
- **WebSocket Forwarding**: All MQTT messages are forwarded to connected WebSocket clients
- **Authentication Support**: Optional username/password authentication
- **Automatic Reconnection**: Handles connection failures and automatically reconnects
- **Structured Message Format**: Messages are formatted with topic, data, and timestamp
## Usage
### Basic Usage
Start the gateway with MQTT integration enabled:
```bash
./spore-gateway -mqtt tcp://localhost:1883
```
### With Authentication
If your MQTT broker requires authentication, use environment variables:
```bash
MQTT_USER=username MQTT_PASSWORD=password ./spore-gateway -mqtt tcp://broker.example.com:1883
```
### Complete Example
```bash
# Terminal 1: Start MQTT broker (optional, for testing)
cd hack
./mosquitto.sh
# Terminal 2: Start SPORE gateway with MQTT integration
cd ..
MQTT_USER=admin MQTT_PASSWORD=secret ./spore-gateway -mqtt tcp://localhost:1883
# Terminal 3: Publish test messages
cd hack
./mqtt-test.sh
```
## Message Format
MQTT messages received by the gateway are forwarded to WebSocket clients with the following JSON structure:
```json
{
"topic": "sensor/temperature/living-room",
"data": "{\"temperature\": 23.5, \"unit\": \"celsius\", \"timestamp\": \"2024-01-15T10:30:00Z\"}",
"timestamp": "2024-01-15T10:30:00Z"
}
```
### Fields
- **topic** (string): The MQTT topic the message was published to
- **data** (string): The raw message payload as a string (can be JSON, text, binary data encoded as string, etc.)
- **timestamp** (string): RFC3339 timestamp when the gateway received the message
### Message Payload Handling
The gateway treats all MQTT message payloads as raw data (byte arrays). When forwarding to WebSocket:
- Binary data is converted to string representation
- Text data is forwarded as-is
- JSON data remains as JSON string (not parsed)
This preserves the original message format while allowing the WebSocket client to parse or display it as needed.
## Configuration
### Command Line Flags
| Flag | Description | Example |
|------|-------------|---------|
| `-mqtt` | MQTT broker URL | `tcp://localhost:1883` |
### Environment Variables
| Variable | Description | Required |
|----------|-------------|----------|
| `MQTT_USER` | Username for MQTT authentication | No |
| `MQTT_PASSWORD` | Password for MQTT authentication | No |
### Broker URLs
Supported URL formats:
- `tcp://hostname:port` - Standard MQTT (e.g., `tcp://localhost:1883`)
- `tcp://hostname` - Uses default port 1883
- `tls://hostname:8883` - Secure MQTT with TLS
Note: TLS support may require additional configuration in the MQTT client.
## Architecture
### Components
1. **MQTT Client** (`internal/mqtt/mqtt.go`)
- Manages connection to MQTT broker
- Handles subscriptions and message reception
- Implements reconnection logic
2. **WebSocket Server** (`internal/websocket/websocket.go`)
- Broadcasting MQTT messages to connected clients
- Serialization and message formatting
3. **Main Application** (`main.go`)
- Coordinates MQTT client initialization
- Sets up message callback for WebSocket forwarding
### Data Flow
```
MQTT Broker → MQTT Client → Callback → HTTP Server → WebSocket Server → Client
```
1. MQTT broker publishes message to any topic
2. Gateway's MQTT client receives message
3. Message callback triggers
4. HTTP server broadcasts to WebSocket
5. WebSocket server forwards to all connected clients
## Testing
### Local Testing Setup
The `hack/` directory contains scripts for testing MQTT integration:
```bash
# Start a local MQTT broker
./hack/mosquitto.sh
# Run comprehensive test suite
./hack/mqtt-test.sh
```
### Test Messages
The test suite includes 16 different message types:
- Simple text messages
- JSON sensor data (temperature, humidity)
- Device status updates
- System events and alerts
- Configuration updates
- Metrics and telemetry
- Node discovery events
- Firmware updates
- Task status
- Error logs
- Light control (SPORE-specific)
- Binary data
- Edge cases (empty messages, large payloads)
### Manual Testing
You can also publish messages manually using the Mosquitto client:
```bash
# Install mosquitto clients
# Ubuntu/Debian: apt-get install mosquitto-clients
# Or use Docker: docker run --rm -it --network host eclipse-mosquitto:latest mosquitto_pub
# Publish a test message
docker run --rm --network host eclipse-mosquitto:latest \
mosquitto_pub -h localhost -p 1883 -t "test/topic" -m "Hello World"
# Publish JSON message
docker run --rm --network host eclipse-mosquitto:latest \
mosquitto_pub -h localhost -p 1883 -t "sensor/data" \
-m '{"sensor": "temperature", "value": 25.5, "unit": "celsius"}'
```
## Integration with SPORE UI
When the SPORE UI connects to the gateway's WebSocket endpoint, it will automatically receive MQTT messages. The UI can handle these messages similarly to SPORE cluster events.
### WebSocket Event Types
The WebSocket receives different event types:
- **Cluster Events**: `cluster/update`, `node/discovery`, etc. (from SPORE nodes)
- **MQTT Events**: Any topic from MQTT (identified by the topic field)
Example WebSocket message from MQTT:
```json
{
"topic": "sensor/temperature",
"data": "23.5",
"timestamp": "2024-01-15T10:30:00Z"
}
```
Example WebSocket message from SPORE cluster:
```json
{
"topic": "cluster/update",
"members": [...],
"primaryNode": "192.168.1.100",
"totalNodes": 3,
"timestamp": "2024-01-15T10:30:00Z"
}
```
## Troubleshooting
### Connection Issues
**Problem**: Gateway fails to connect to MQTT broker
**Solutions**:
```bash
# Check if broker is running
docker ps | grep mqtt-broker
# Check broker logs
docker logs mqtt-broker
# Test connection manually
docker run --rm -it --network host eclipse-mosquitto:latest \
mosquitto_pub -h localhost -p 1883 -t "test" -m "test"
```
### Messages Not Forwarding
**Problem**: MQTT messages not appearing in WebSocket
**Solutions**:
1. Verify gateway is running with MQTT enabled
2. Check gateway logs for MQTT connection status
3. Verify WebSocket client is connected
4. Check MQTT broker logs for subscription confirmation
### Authentication Errors
**Problem**: "Connection refused" or authentication errors
**Solutions**:
```bash
# Ensure environment variables are set
export MQTT_USER=username
export MQTT_PASSWORD=password
# Verify broker allows connections
# Check mosquitto.conf for allow_anonymous or authentication settings
```
### High Message Volume
If receiving many MQTT messages:
- Gateway handles messages efficiently using Go concurrency
- WebSocket broadcasts are serialized to prevent race conditions
- Consider QoS levels if message delivery is critical
## Best Practices
### Topic Naming
Use hierarchical topic names for better organization:
```
sensor/temperature/living-room
sensor/humidity/bedroom
device/status/esp32-001
cluster/node/discovered
```
### Message Size
- Keep individual messages reasonably sized (< 10KB recommended)
- For large data, consider splitting into multiple messages
- Use compression if transmitting large JSON payloads
### Security
- Use authentication for production deployments
- Consider TLS for encrypted connections
- Use topic filtering if subscribing to specific topics only (modify subscription)
- Implement rate limiting on message processing if needed
### Error Handling
The gateway includes automatic reconnection logic:
- Initial connection failures are logged
- Reconnection attempts every 10 seconds
- Connection state is tracked and logged
- WebSocket clients are notified via disconnect events
## Limitations
- **QoS Levels**: Currently uses QoS 0 (at most once delivery)
- **Topic Filtering**: Subscribes to all topics (`#`); no selective subscription
- **Message Retention**: Does not store messages; forwards only real-time events
- **Duplicate Handling**: Does not deduplicate messages
- **Ordering**: Maintains message order within individual WebSocket broadcasts
## Future Enhancements
Potential improvements:
- Configurable QoS levels per topic
- Selective topic subscription via configuration
- Message persistence and replay
- Metrics and monitoring for MQTT integration
- Support for MQTT 5.0 features
## Related Documentation
- [Main README](../README.md) - Overview of SPORE Gateway
- [Rollout Documentation](./Rollout.md) - Firmware rollout process
- [Hack Directory](../hack/README.md) - Testing tools and scripts
## Examples
### Example: IoT Sensor Integration
Connect temperature sensors to the gateway:
```bash
# Start gateway with MQTT
./spore-gateway -mqtt tcp://iot-broker.example.com:1883
# Sensors publish to topics like:
# - sensor/temperature/room1
# - sensor/humidity/room1
# - sensor/light/room1
```
### Example: Device Control
Control SPORE nodes via MQTT:
```bash
# Publish control commands
mosquitto_pub -h broker.example.com -t "spore/control" \
-m '{"node": "esp32-001", "action": "pattern", "pattern": "rainbow"}'
```
### Example: Monitoring Dashboard
Combine SPORE cluster events with external system events:
```bash
# Gateway receives both:
# 1. SPORE cluster events (from UDP discovery)
# 2. External system events (via MQTT)
# UI displays unified event stream
```
## Support
For issues or questions about MQTT integration:
- Check gateway logs for MQTT connection status
- Review MQTT broker configuration
- Use `hack/mqtt-test.sh` for testing
- See [troubleshooting section](#troubleshooting) above

73
docs/README.md Normal file
View File

@@ -0,0 +1,73 @@
# SPORE Gateway Documentation
Welcome to the SPORE Gateway documentation. This directory contains detailed documentation for various features and capabilities of the gateway.
## Available Documentation
### [MQTT Integration](./MQTT.md)
Comprehensive guide to the MQTT integration feature, including:
- Setting up MQTT integration
- Message format and handling
- Testing with local MQTT brokers
- Architecture and data flow
- Troubleshooting and best practices
### [Rollout Process](./Rollout.md)
Detailed documentation for the firmware rollout system:
- Parallel firmware updates across multiple nodes
- WebSocket progress updates
- Integration with spore-registry
- API endpoints and message formats
## Quick Links
- **Main README**: [../README.md](../README.md)
- **Hack Directory**: [../hack/README.md](../hack/README.md)
- **Testing Scripts**: [../hack/](../hack/)
## Feature Overview
### Core Features
- UDP-based node discovery
- Cluster management and primary node selection
- HTTP API server for cluster operations
- WebSocket real-time updates
- Failover logic for automatic primary switching
- Generic proxy calls to SPORE nodes
### Integration Features
- **MQTT Integration**: Subscribe to MQTT topics and forward messages to WebSocket clients
- **Firmware Rollout**: Orchestrated firmware updates across the cluster
- **Registry Proxy**: Proxy for spore-registry firmware management
## Getting Started
1. **Basic Setup**: See [Main README](../README.md) for installation and basic usage
2. **MQTT Integration**: See [MQTT.md](./MQTT.md) for MQTT setup and testing
3. **Testing**: See [Hack README](../hack/README.md) for local testing tools
## Development
The gateway is written in Go and follows modern Go best practices:
- Structured logging using logrus
- Graceful shutdown handling
- Concurrent-safe operations
- HTTP middleware for CORS and logging
- WebSocket support for real-time updates
## Contributing
When adding new features:
1. Update relevant documentation in this directory
2. Add examples to the `hack/` directory
3. Update the main README with feature highlights
4. Follow the existing documentation style and structure
## Support
For questions or issues:
- Check the relevant documentation in this directory
- Review gateway logs for error messages
- Use testing tools in the `hack/` directory
- Check the main README for troubleshooting tips