This is an early status report on subject R&D project, aka “WUWH” for “Wish You Were Here”.

The goal is a real-time multi-user “holodeck”-like system where avatars of users and their surroundings share a 3D mixed-reality environment. Development is to proceed on two fronts: object-tracking, and VR rendering.
Approach:
- TensorFlow Lite does object detection on RPi
- Coral TPU speeds TF processing (from 2 to 11 FPS x 2 cameras)
- OpenCV with some custom programming for stereo depth detection
- 3D object tracks sent UDP as JSON packets to CloudTurbine on PC/Mac
- CTrollaball for live/playback view of 3D results over network
Features:
- Distributed participants
- Monitor, record, playback
- Modular data sources, middleware, viewers
Key Elements:
- Raspberry Pi 4
- TensorFlow Lite
- Coral TPU
- Dual USB Webcams
- CloudTurbine Multi-Spatial Recorder
- CTrollaball V1.1 viewer
- Custom system integration software

