spore-gateway/docs/MQTT.md

# MQTT Integration

The SPORE Gateway includes optional MQTT integration that allows subscribing to MQTT brokers and forwarding messages to connected WebSocket clients. This enables integration with IoT devices, sensor networks, and other MQTT-based systems.

## Overview

When enabled, the gateway acts as an MQTT subscriber that:
- Connects to an MQTT broker
- Subscribes to all topics (`#`)
- Forwards received messages to WebSocket clients in real-time

This allows the SPORE UI to display MQTT events alongside SPORE cluster events.

## Features

- **Universal Topic Subscription**: Subscribes to `#` (all topics) to capture all messages
- **WebSocket Forwarding**: All MQTT messages are forwarded to connected WebSocket clients
- **Authentication Support**: Optional username/password authentication
- **Automatic Reconnection**: Handles connection failures and automatically reconnects
- **Structured Message Format**: Messages are formatted with topic, data, and timestamp

## Usage

### Basic Usage

Start the gateway with MQTT integration enabled:

```bash
./spore-gateway -mqtt tcp://localhost:1883
```

### With Authentication

If your MQTT broker requires authentication, use environment variables:

```bash
MQTT_USER=username MQTT_PASSWORD=password ./spore-gateway -mqtt tcp://broker.example.com:1883
```

### Complete Example

```bash
# Terminal 1: Start MQTT broker (optional, for testing)
cd hack
./mosquitto.sh

# Terminal 2: Start SPORE gateway with MQTT integration
cd ..
MQTT_USER=admin MQTT_PASSWORD=secret ./spore-gateway -mqtt tcp://localhost:1883

# Terminal 3: Publish test messages
cd hack
./mqtt-test.sh
```

## Message Format

MQTT messages received by the gateway are forwarded to WebSocket clients with the following JSON structure:

```json
{
  "topic": "sensor/temperature/living-room",
  "data": "{\"temperature\": 23.5, \"unit\": \"celsius\", \"timestamp\": \"2024-01-15T10:30:00Z\"}",
  "timestamp": "2024-01-15T10:30:00Z"
}
```

### Fields

- **topic** (string): The MQTT topic the message was published to
- **data** (string): The raw message payload as a string (can be JSON, text, binary data encoded as string, etc.)
- **timestamp** (string): RFC3339 timestamp when the gateway received the message

### Message Payload Handling

The gateway treats all MQTT message payloads as raw data (byte arrays). When forwarding to WebSocket:
- Binary data is converted to string representation
- Text data is forwarded as-is
- JSON data remains as JSON string (not parsed)

This preserves the original message format while allowing the WebSocket client to parse or display it as needed.

## Configuration

### Command Line Flags

| Flag | Description | Example |
|------|-------------|---------|
| `-mqtt` | MQTT broker URL | `tcp://localhost:1883` |

### Environment Variables

| Variable | Description | Required |
|----------|-------------|----------|
| `MQTT_USER` | Username for MQTT authentication | No |
| `MQTT_PASSWORD` | Password for MQTT authentication | No |

### Broker URLs

Supported URL formats:
- `tcp://hostname:port` - Standard MQTT (e.g., `tcp://localhost:1883`)
- `tcp://hostname` - Uses default port 1883
- `tls://hostname:8883` - Secure MQTT with TLS

Note: TLS support may require additional configuration in the MQTT client.

## Architecture

### Components

1. **MQTT Client** (`internal/mqtt/mqtt.go`)
   - Manages connection to MQTT broker
   - Handles subscriptions and message reception
   - Implements reconnection logic

2. **WebSocket Server** (`internal/websocket/websocket.go`)
   - Broadcasting MQTT messages to connected clients
   - Serialization and message formatting

3. **Main Application** (`main.go`)
   - Coordinates MQTT client initialization
   - Sets up message callback for WebSocket forwarding

### Data Flow

```
MQTT Broker → MQTT Client → Callback → HTTP Server → WebSocket Server → Client
```

1. MQTT broker publishes message to any topic
2. Gateway's MQTT client receives message
3. Message callback triggers
4. HTTP server broadcasts to WebSocket
5. WebSocket server forwards to all connected clients

## Testing

### Local Testing Setup

The `hack/` directory contains scripts for testing MQTT integration:

```bash
# Start a local MQTT broker
./hack/mosquitto.sh

# Run comprehensive test suite
./hack/mqtt-test.sh
```

### Test Messages

The test suite includes 16 different message types:
- Simple text messages
- JSON sensor data (temperature, humidity)
- Device status updates
- System events and alerts
- Configuration updates
- Metrics and telemetry
- Node discovery events
- Firmware updates
- Task status
- Error logs
- Light control (SPORE-specific)
- Binary data
- Edge cases (empty messages, large payloads)

### Manual Testing

You can also publish messages manually using the Mosquitto client:

```bash
# Install mosquitto clients
# Ubuntu/Debian: apt-get install mosquitto-clients
# Or use Docker: docker run --rm -it --network host eclipse-mosquitto:latest mosquitto_pub

# Publish a test message
docker run --rm --network host eclipse-mosquitto:latest \
  mosquitto_pub -h localhost -p 1883 -t "test/topic" -m "Hello World"

# Publish JSON message
docker run --rm --network host eclipse-mosquitto:latest \
  mosquitto_pub -h localhost -p 1883 -t "sensor/data" \
  -m '{"sensor": "temperature", "value": 25.5, "unit": "celsius"}'
```

## Integration with SPORE UI

When the SPORE UI connects to the gateway's WebSocket endpoint, it will automatically receive MQTT messages. The UI can handle these messages similarly to SPORE cluster events.

### WebSocket Event Types

The WebSocket receives different event types:

- **Cluster Events**: `cluster/update`, `node/discovery`, etc. (from SPORE nodes)
- **MQTT Events**: Any topic from MQTT (identified by the topic field)

Example WebSocket message from MQTT:

```json
{
  "topic": "sensor/temperature",
  "data": "23.5",
  "timestamp": "2024-01-15T10:30:00Z"
}
```

Example WebSocket message from SPORE cluster:

```json
{
  "topic": "cluster/update",
  "members": [...],
  "primaryNode": "192.168.1.100",
  "totalNodes": 3,
  "timestamp": "2024-01-15T10:30:00Z"
}
```

## Troubleshooting

### Connection Issues

**Problem**: Gateway fails to connect to MQTT broker

**Solutions**:
```bash
# Check if broker is running
docker ps | grep mqtt-broker

# Check broker logs
docker logs mqtt-broker

# Test connection manually
docker run --rm -it --network host eclipse-mosquitto:latest \
  mosquitto_pub -h localhost -p 1883 -t "test" -m "test"
```

### Messages Not Forwarding

**Problem**: MQTT messages not appearing in WebSocket

**Solutions**:
1. Verify gateway is running with MQTT enabled
2. Check gateway logs for MQTT connection status
3. Verify WebSocket client is connected
4. Check MQTT broker logs for subscription confirmation

### Authentication Errors

**Problem**: "Connection refused" or authentication errors

**Solutions**:
```bash
# Ensure environment variables are set
export MQTT_USER=username
export MQTT_PASSWORD=password

# Verify broker allows connections
# Check mosquitto.conf for allow_anonymous or authentication settings
```

### High Message Volume

If receiving many MQTT messages:
- Gateway handles messages efficiently using Go concurrency
- WebSocket broadcasts are serialized to prevent race conditions
- Consider QoS levels if message delivery is critical

## Best Practices

### Topic Naming

Use hierarchical topic names for better organization:
```
sensor/temperature/living-room
sensor/humidity/bedroom
device/status/esp32-001
cluster/node/discovered
```

### Message Size

- Keep individual messages reasonably sized (< 10KB recommended)
- For large data, consider splitting into multiple messages
- Use compression if transmitting large JSON payloads

### Security

- Use authentication for production deployments
- Consider TLS for encrypted connections
- Use topic filtering if subscribing to specific topics only (modify subscription)
- Implement rate limiting on message processing if needed

### Error Handling

The gateway includes automatic reconnection logic:
- Initial connection failures are logged
- Reconnection attempts every 10 seconds
- Connection state is tracked and logged
- WebSocket clients are notified via disconnect events

## Limitations

- **QoS Levels**: Currently uses QoS 0 (at most once delivery)
- **Topic Filtering**: Subscribes to all topics (`#`); no selective subscription
- **Message Retention**: Does not store messages; forwards only real-time events
- **Duplicate Handling**: Does not deduplicate messages
- **Ordering**: Maintains message order within individual WebSocket broadcasts

## Future Enhancements

Potential improvements:
- Configurable QoS levels per topic
- Selective topic subscription via configuration
- Message persistence and replay
- Metrics and monitoring for MQTT integration
- Support for MQTT 5.0 features

## Related Documentation

- [Main README](../README.md) - Overview of SPORE Gateway
- [Rollout Documentation](./Rollout.md) - Firmware rollout process
- [Hack Directory](../hack/README.md) - Testing tools and scripts

## Examples

### Example: IoT Sensor Integration

Connect temperature sensors to the gateway:

```bash
# Start gateway with MQTT
./spore-gateway -mqtt tcp://iot-broker.example.com:1883

# Sensors publish to topics like:
# - sensor/temperature/room1
# - sensor/humidity/room1
# - sensor/light/room1
```

### Example: Device Control

Control SPORE nodes via MQTT:

```bash
# Publish control commands
mosquitto_pub -h broker.example.com -t "spore/control" \
  -m '{"node": "esp32-001", "action": "pattern", "pattern": "rainbow"}'
```

### Example: Monitoring Dashboard

Combine SPORE cluster events with external system events:

```bash
# Gateway receives both:
# 1. SPORE cluster events (from UDP discovery)
# 2. External system events (via MQTT)

# UI displays unified event stream
```

## Support

For issues or questions about MQTT integration:
- Check gateway logs for MQTT connection status
- Review MQTT broker configuration
- Use `hack/mqtt-test.sh` for testing
- See [troubleshooting section](#troubleshooting) above