Augmentation of Real-Time Registered Surfaces using a Projector-3D Camera System - Raul Mircea Gigea

Projector-Camera systems are the main building block for Spatial Augmented Reality(SAR) setups. SAR is concerned with visually enhancing the physical reality with virtual generated information by using the environment as a display surface for the projector. Such systems are characterized, as opposed to simple AR, by the spatial relationship (registration) between the display surface and the projected information.

This thesis presents a specialized SAR system which is able to augment arbitrary, dynamic scenes in real-time without prior knowledge of the environment, and able to use the information about the environment for geometry-aware augmentation. We do this by extending a SAR system to use flexible real-time acquired depth information from a time-of-flight photonic mixing device (PMD). This information is needed to react to environment changes and maintain the spatial relationship with the displayed image.

One of the main challenges of SAR setups is modeling and calibrating the projector-to-world registration. Our system setup enables the automatic calibration of the projector-to-world model using closed-loop feedback from a stereo camera and previously calibrated stereo- and PMD camera. The calibrated model is then used with the PMD data to register the display surface with the environment in real-time.

PMD setups need extensive, manual calibration of the depth values in order to deliver accurate results. Our framework implements an automatic depth calibration using reference values delivered by the stereo camera through common projector assisted scanning.

In an example application we demonstrate how this framework can be used to derive the boundaries of moving objects, such as a person, and reproject them as a visual augmentation in real-time. Typical PMD technology limitations in the context of SAR setups are presented.