This is an early status report on subject R&D project, aka “WUWH” for “Wish You Were Here”.

Development system with RPi, TPU, depth-perception stereo webcams

The goal is a real-time multi-user “holodeck”-like system where avatars of users and their surroundings share a 3D mixed-reality environment. Development is to proceed on two fronts: object-tracking, and VR rendering.

Approach:

  • TensorFlow Lite does object detection on RPi
  • Coral TPU speeds TF processing (from 2 to 11 FPS x 2 cameras)
  • OpenCV with some custom programming for stereo depth detection
  • 3D object tracks sent UDP as JSON packets to CloudTurbine on PC/Mac
  • CTrollaball for live/playback view of 3D results over network

Features:

  • Distributed participants
  • Monitor, record, playback
  • Modular data sources, middleware, viewers

Key Elements:

  • Raspberry Pi 4
  • TensorFlow Lite
  • Coral TPU
  • Dual USB Webcams
  • CloudTurbine Multi-Spatial Recorder
  • CTrollaball V1.1 viewer
  • Custom system integration software
Stereo camera object detection at 11 FPS (x2 cams) using RPi4 with Coral TPU

Live “CTrollaball” 3D Unity viewer with dog and vase detected (3D spheres)