I showed that it worked and that I can take photos with what is basically a Raspberry Beowulf Cluster.
Didn't last long...
Another point is the chassis for the at this point dual cam system, having the raspis run hot over time, which still needs way too much duck tape. Version 4 is already created in the Printer Software and needs to manifest in the physical world now.
From here, a Summery of the new version with a full path, part of the Cyberdeck Project:
------------------
Below is a polished, self‑contained project manual for your website. It covers the design, the “why”, and a full walkthrough – perfect for others to replicate or for you to refer back to.
---
# SmartCam: a remote‑controlled dual‑camera cluster appliance
**A travel‑ready AI camera system built on a two‑node Beowulf cluster with a Waveshare screen, Hailo‑8L AI accelerator, and GPRS connectivity.**
---
## 1. Overview
The SmartCam is a head‑and‑sensor cluster designed for road trips, landmark identification, and real‑time AI‑assisted driving.
It consists of two Raspberry Pi 5 nodes:
- **Node1 (head)** – DietPi, 8 GB RAM, HQ camera, Waveshare 3.5″ SPI display, Hailo‑8L AI kit, GPRS module.
- **Node7 (night sensor)** – Raspberry Pi OS Lite 64‑bit, 16 GB RAM, NoIR camera.
Both nodes are part of a larger Beowulf cluster and communicate through a dedicated **mpiuser** SSH fabric. The head node can **locally** preview the HQ camera or **remotely** pull the NoIR camera’s video stream via an SSH tunnel, displaying it directly on the Waveshare screen – all without a desktop environment, controllable by a local keyboard or by any SSH session in the cluster.
This document explains the system design, how each piece was built, and how to set it up from scratch.
---
## 2. Design philosophy
- **No X11, no desktop** – the Waveshare screen is written to directly via the Linux framebuffer (`/dev/fb0`). This is fast, lightweight, and crash‑resistant.
- **One communication layer** – all inter‑node traffic goes through the pre‑existing `mpiuser` SSH keys and the cluster‑wide `/etc/hosts`. No new passwords, no open ports.
- **Modular daemons** – each function (camera capture, stream fetching, display rendering, keyboard input, remote commands) runs in its own thread inside a single, restartable service.
- **Remote‑first** – the entire system is operated from an SSH terminal. The display can be toggled between cameras with a single command.
---
## 3. Hardware and wiring
| Component | Connected to | Notes
|-------------------------------|-----------------------------|----------------------------------------
| HQ camera | Node1 CSI | IMX477 sensor, visible light
| NoIR camera | Node7 CSI | OV5647 sensor, infrared‑sensitive
| Waveshare 3.5″ SPI | Node1 GPIO | ILI9486 controller, 480×320, 16‑bit
| Hailo‑8L | Node1 M.2 HAT | PCIe, used for future AI inference
| GPRS module | Node1 UART | Serial AT commands (future SMS alerts)
| Keyboard | Node1 USB | Used to toggle camera feed with F3
---
## 4. Software architecture
```
┌─────────────────────┐
│ node1 (head) │
│ │
│ smartcam-displayd │
│ ┌───────────────┐ │
│ │ HQCamera │ │ picamera2 → HQ frames
│ ├───────────────┤ │
│ │ NoirStream │ ──┼── HTTP over SSH tunnel → node7:5000
│ ├───────────────┤ │
│ │ Display │ ──┼── writes to /dev/fb0
│ ├───────────────┤ │
│ │ Keyboard │ │ evdev → F3 toggles source
│ ├───────────────┤ │
│ │ UnixSocket │ │ /run/smartcam/control.sock
│ └───────────────┘ │
│ │
│ smartcam-tunnel │ autossh -L 5099:localhost:5000 node7
└─────────┬───────────┘
│ mpiuser SSH
▼
┌─────────────────────┐
│ node7 (night) │
│ │
│ noir-streamer │ Flask MJPEG server on 127.0.0.1:5000
│ (picamera2) │
└─────────────────────┘
```
**Key flows:**
1. **node7** continuously captures NoIR frames and serves them as an MJPEG stream on `localhost:5000`.
2. **node1** runs an `autossh` tunnel that forwards its own `localhost:5099` to node7’s `localhost:5000`. The tunnel uses the `mpiuser` identity and reconnects automatically.
3. **node1**’s main daemon (`smartcam-displayd`):
- Captures HQ frames locally via `picamera2`.
- Reads the tunnelled MJPEG stream from `localhost:5099`.
- Maintains a shared active source (`hq` or `noir`).
- Continuously renders the selected source onto the Waveshare framebuffer.
- Listens for `F3` keypresses on a USB keyboard to toggle the source.
- Creates a Unix socket for remote commands (`switch hq`, `switch noir`, `status`, `stop`).
---
## 5. Setup instructions
### 5.1 Prerequisites
- Both Pis flashed with their respective OS images (DietPi on node1, Raspberry Pi OS Lite 64‑bit on node7).
- Cameras enabled (`raspi-config` or `dietpi-config`), SPI enabled on node1.
- Static IP addresses assigned:
- **node1**: 192.168.178.30
- **node7**: 192.168.178.39
- The cluster‑wide `/etc/hosts` on both Pis must contain:
```
192.168.178.30 node1 raspi5
192.168.178.39 node7 16gbraspi
```
- The `mpiuser` account exists on both nodes with SSH key‑based authentication set up.
### 5.2 node7 – NoIR MJPEG streamer
1. **Install dependencies** (if not already present):
```bash
sudo apt update
sudo apt install python3-picamera2 python3-flask python3-opencv
```
2. **Create the streamer script** at `/home/mpiuser/noir_stream.py`:
```python
#!/usr/bin/env python3
from flask import Flask, Response
import picamera2
import cv2
import threading
import time
app = Flask(__name__)
camera = None
output_jpeg = None
lock = threading.Lock()
def init_camera():
global camera
camera = picamera2.Picamera2()
config = camera.create_video_configuration(main={"size": (640, 480)})
camera.configure(config)
camera.start()
time.sleep(1)
def capture_loop():
global output_jpeg
while True:
frame = camera.capture_array()
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
ret, jpeg = cv2.imencode('.jpg', rgb)
with lock:
output_jpeg = jpeg.tobytes()
time.sleep(0.05)
@app.route('/stream.mjpg')
def video_feed():
def generate():
while True:
with lock:
if output_jpeg is not None:
yield (b'--frame\r\n'
b'Content-Type: image/jpeg\r\n\r\n' + output_jpeg + b'\r\n')
time.sleep(0.05)
return Response(generate(), mimetype='multipart/x-mixed-replace; boundary=frame')
if __name__ == '__main__':
init_camera()
threading.Thread(target=capture_loop, daemon=True).start()
app.run(host='127.0.0.1', port=5000, debug=False)
```
3. **Make it a systemd service** (`/etc/systemd/system/noir-streamer.service`):
```
[Unit]
Description=NoIR Camera MJPEG Streamer
After=network.target dev-media1.device
Wants=dev-media1.device
[Service]
ExecStartPre=/bin/sleep 2
ExecStart=/usr/bin/python3 /home/mpiuser/noir_stream.py
Restart=always
User=mpiuser
Group=mpiuser
[Install]
WantedBy=multi-user.target
```
4. **Enable and start**:
```bash
sudo systemctl daemon-reload
sudo systemctl enable noir-streamer
sudo systemctl start noir-streamer
```
5. **Verify**:
```bash
curl http://localhost:5000/stream.mjpg
```
You should see a continuous stream of binary JPEG frames.
### 5.3 node1 – SSH tunnel to node7
1. **Install autossh**:
```bash
sudo apt install autossh
```
2. **Create the tunnel service** (`/etc/systemd/system/smartcam-tunnel.service`):
```
[Unit]
Description=SmartCam Night Stream Tunnel
After=network-online.target
Wants=network-online.target
[Service]
User=mpiuser
ExecStart=/usr/bin/autossh -M 0 -N -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -L 5099:localhost:5000 node7
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
3. **Ensure mpiuser can SSH to node7 without password** (run once):
```bash
sudo -u mpiuser ssh -o StrictHostKeyChecking=accept-new mpiuser@node7 echo ok
```
4. **Start and test**:
```bash
sudo systemctl enable smartcam-tunnel
sudo systemctl start smartcam-tunnel
curl http://localhost:5099/stream.mjpg # should produce JPEG stream
```
### 5.4 node1 – Display daemon
1. **Install Python dependencies** (on DietPi, package names may vary slightly):
```bash
sudo apt update
sudo apt install python3-requests python3-picamera2 python3-pil python3-numpy python3-opencv python3-evdev
```
2. **Disable any display manager** (the daemon uses the raw framebuffer):
```bash
sudo systemctl stop lightdm # or whatever DM DietPi uses
sudo systemctl disable lightdm
```
After a reboot, you should only have a text console. The Waveshare screen may stay black – that’s fine.
3. **Identify the framebuffer device** – for the ILI9486, it’s `/dev/fb0`. Confirm with:
```bash
cat /sys/class/graphics/fb0/name # should output "fb_ili9486"
```
4. **Create the daemon script** at `/usr/local/bin/smartcam-displayd`:
(Use the full script from the final version below – it includes manual RGB565 packing to avoid Pillow encoder issues on DietPi.)
```python
#!/usr/bin/env python3
import os, mmap, time, threading, socket, signal
import numpy as np
import cv2
from picamera2 import Picamera2
import requests
import evdev
from evdev import ecodes
from PIL import Image
# ------ Configuration ------
FRAMEBUFFER_DEV = '/dev/fb0'
SCREEN_WIDTH = 480
SCREEN_HEIGHT = 320
NOIR_STREAM_URL = 'http://localhost:5099/stream.mjpg'
KEYBOARD_DEV = '/dev/input/by-id/usb-Logitech_USB_Keyboard-event-kbd' # adjust to your keyboard
CONTROL_SOCKET = '/run/smartcam/control.sock'
# ---------------------------
class State:
def __init__(self):
self.lock = threading.Lock()
self.active_source = 'hq'
self.hq_frame = None
self.noir_frame = None
self.running = True
state = State()
# Open framebuffer
fb_fd = os.open(FRAMEBUFFER_DEV, os.O_RDWR)
buf_size = SCREEN_WIDTH * SCREEN_HEIGHT * 2 # 16 bits per pixel
fb_buf = mmap.mmap(fb_fd, buf_size, mmap.MAP_SHARED, mmap.PROT_WRITE)
def write_frame_to_fb(rgb_image):
"""Convert RGB numpy array to RGB565 and write to framebuffer."""
img = Image.fromarray(rgb_image).resize((SCREEN_WIDTH, SCREEN_HEIGHT))
arr = np.array(img, dtype=np.uint8)
r = (arr[:,:,0] >> 3).astype(np.uint16)
g = (arr[:,:,1] >> 2).astype(np.uint16)
b = (arr[:,:,2] >> 3).astype(np.uint16)
packed = (r << 11) | (g << 5) | b
fb_buf.seek(0)
fb_buf.write(packed.tobytes())
class HQCamera(threading.Thread):
def __init__(self):
super().__init__(daemon=True)
self.picam2 = Picamera2()
config = self.picam2.create_video_configuration(main={"size": (640, 480)})
self.picam2.configure(config)
def run(self):
self.picam2.start()
time.sleep(1)
while state.running:
frame = self.picam2.capture_array()
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
with state.lock:
state.hq_frame = rgb
time.sleep(0.05)
class NoirStream(threading.Thread):
def __init__(self):
super().__init__(daemon=True)
def run(self):
while state.running:
try:
r = requests.get(NOIR_STREAM_URL, stream=True, timeout=2.0)
buf = b''
for chunk in r.iter_content(chunk_size=1024):
if not state.running: break
buf += chunk
a = buf.find(b'\xff\xd8')
b = buf.find(b'\xff\xd9')
if a != -1 and b != -1:
jpg = buf[a:b+2]
buf = buf[b+2:]
img = cv2.imdecode(np.frombuffer(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
if img is not None:
rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
with state.lock:
state.noir_frame = rgb
except Exception as e:
print(f"Noir stream error: {e}")
time.sleep(1)
class Display(threading.Thread):
def __init__(self):
super().__init__(daemon=True)
def run(self):
while state.running:
with state.lock:
source = state.active_source
frame = state.hq_frame if source == 'hq' else state.noir_frame
if frame is not None:
write_frame_to_fb(frame)
time.sleep(0.05)
class Keyboard(threading.Thread):
def __init__(self, dev_path):
super().__init__(daemon=True)
self.dev = evdev.InputDevice(dev_path)
def run(self):
for event in self.dev.read_loop():
if event.type == ecodes.EV_KEY and event.value == 1:
if event.code == ecodes.KEY_F3:
with state.lock:
state.active_source = 'noir' if state.active_source == 'hq' else 'hq'
print(f"Switched to {state.active_source}")
def command_server():
if os.path.exists(CONTROL_SOCKET):
os.remove(CONTROL_SOCKET)
os.makedirs(os.path.dirname(CONTROL_SOCKET), exist_ok=True)
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(CONTROL_SOCKET)
os.chmod(CONTROL_SOCKET, 0o660)
server.listen(1)
server.settimeout(0.5)
while state.running:
try:
conn, _ = server.accept()
data = conn.recv(1024).decode().strip()
if data == 'switch hq':
state.active_source = 'hq'
conn.sendall(b'OK\n')
elif data == 'switch noir':
state.active_source = 'noir'
conn.sendall(b'OK\n')
elif data == 'status':
conn.sendall(f'Active source: {state.active_source}\n'.encode())
elif data == 'stop':
state.running = False
conn.sendall(b'OK: stopping\n')
else:
conn.sendall(b'ERROR\n')
conn.close()
except socket.timeout:
continue
server.close()
os.remove(CONTROL_SOCKET)
if __name__ == '__main__':
# Auto-detect keyboard if hardcoded path doesn't exist
import glob
kbd_path = KEYBOARD_DEV
if not os.path.exists(kbd_path):
for dev in glob.glob('/dev/input/by-id/*-kbd'):
kbd_path = dev
break
print(f"Using keyboard: {kbd_path}")
threads = [
HQCamera(),
NoirStream(),
Display(),
Keyboard(kbd_path),
threading.Thread(target=command_server, daemon=True)
]
for t in threads:
t.start()
signal.pause()
```
5. **Create the systemd unit** (`/etc/systemd/system/smartcam-displayd.service`):
```
[Unit]
Description=SmartCam Display Daemon
After=smartcam-tunnel.service
Wants=smartcam-tunnel.service
RuntimeDirectory=smartcam
[Service]
ExecStart=/usr/bin/python3 /usr/local/bin/smartcam-displayd
Restart=always
User=root
Group=root
[Install]
WantedBy=multi-user.target
```
6. **Enable and start**:
```bash
sudo systemctl daemon-reload
sudo systemctl enable smartcam-displayd
sudo systemctl start smartcam-displayd
```
The Waveshare screen should now show the HQ camera preview.
### 5.5 Remote control script
Create `/usr/local/bin/smartcam-ctl`:
```bash
#!/bin/bash
SOCKET="/run/smartcam/control.sock"
if [ ! -S "$SOCKET" ]; then
echo "Daemon not running"
exit 1
fi
if [ $# -eq 0 ]; then
echo "Usage: $0 {switch hq|switch noir|status|stop}"
exit 1
fi
echo "$*" | socat - UNIX-CONNECT:"$SOCKET"
```
Make it executable:
```bash
sudo chmod +x /usr/local/bin/smartcam-ctl
```
Now any user with root privileges (or via `sudo`) can control the display.
---
## 6. Usage
### Local control
- **F3** on the USB keyboard toggles between **HQ camera** and **NoIR camera**.
- The display updates automatically.
### Remote control (from any SSH session)
```bash
# From your workstation (or another cluster node)
ssh J4v@node1
# Check current source
sudo smartcam-ctl status
# Switch to night vision
sudo smartcam-ctl switch noir
# Switch back to HQ
sudo smartcam-ctl switch hq
# Stop the daemon (if needed)
sudo smartcam-ctl stop
```
You can also view the NoIR stream directly on your local machine by setting up an ad‑hoc tunnel:
```bash
ssh -L 5099:localhost:5000 mpiuser@node7
# Then open http://localhost:5099/stream.mjpg in a browser or VLC.
```
### Starting and stopping the services
```bash
# On node7
sudo systemctl start|stop|restart noir-streamer
# On node1
sudo systemctl start|stop|restart smartcam-tunnel
sudo systemctl start|stop|restart smartcam-displayd
```
All services are enabled to start automatically on boot.
---
## 7. Troubleshooting
|------------------------------------------|--------------------------------------|--------------------
| `ModuleNotFoundError: requests`
| `FileNotFoundError: /dev/fb1`
| Screen stays black after daemon starts
| `curl localhost:5099` – empty reply
| `autossh` fails with “Address already in use”
| Keyboard F3 not working
| Pillow encoder error (`RGB;16`)
---
## 8. Next steps – towards an AI travel companion
This base system is now fully remote‑operable and stable. The next phases include:
- **Hailo‑8L AI integration** – a new thread inside `smartcam-displayd` runs object detection on the HQ camera frames, draws bounding boxes, and logs findings.
- **Video recording with detection logs** – triggered by a new socket command, records a video file and timestamps every detection.
- **GPRS alerting** – on specific AI events (landmark, traffic jam), send an SMS via the serial‑attached SIM800 module.
- **Local RAG travel guide** – a quantised LLM runs on node7’s 16 GB RAM, fed with guidebook PDFs; node1 queries it on demand and overlays answers on the screen.
All these extensions use the same SSH tunnel and Unix socket architecture, preserving security and simplicity.
---
*Built with a Beowulf‑style mpiuser communication layer, DietPi on the head, and Raspberry Pi OS Lite on the sensor node. No cloud, no external dependencies – everything runs on the edge.*
