week 5 summary – Hitesh

Kinect Fusion

Kinect Fusion is one of the latest technological advancements in AR. It uses depth data from Kinect sensor to track 3D pose and constructs a 3D representation of the object in the real time. It vouches to be a cost-effective and seamless augmentation of 3D physical data on the real world. As opposed to the conventional technique of creating mesh with depth data which tend to be noisy, Kinect Fusion aims at fusing different viewpoints of the physical scene to create a real 3D model. It works on full depth maps as opposed to Color + depth measuring technique, making it more effective in variable lighting conditions.

A key aspect of Kinect fusion is to support real time user interaction with camera tracking and 3D reconstruction of the physical world. It uses an iterative processing through a raycasted view of the model, which acts as a reference frame for the next iteration rather than just the next depth map frame.  All depth measurements as image coordinates are integrated to a 3D view using volumetric representation. Reconstruction becomes more detailed as depths are added to the measurement. A prominent error with the system is inability to track large and continuous (fast) motions in camera causing the background scene to break.

The most interesting feature of KinectFusion is the ability of the virtual objects to interact dynamically with reconstructed scene in real time through collisions. The geometry aware reconstruction and occlusion handling allows shadowing and lighting, simulating real world physics interaction. These capabilities have a lot of scope in AR applications.


Going Out

Going Out primarily focuses on the limitations of point based tracking in Hand held and other wearable AR computing systems. They are suitable for indoor and small scaled systems but inefficient for large outdoor and complex geometries. Also, other positioning systems such as GPS systems deteriorate in urban environments due to loss of signal quality, and inertial sensors are prone to drift and magnetic sensors.

GoingOut employs a textured 3D model for tracking, which poses the ability to track detailed edges. The system uses edge based trackers with inertial sensors, for accurate results in fast motion in the foreground such as vehicles. The edge search is conducted using an appearance-based model of the edge instead of a simple edge detector. The sensors provide gyroscopic measurements, relative velocity and 3D magnetic field vector which are combined with optical tracking component to make the system more robust and useful in outdoor tracking.  The system uses a recovery model during failure in tracking, wherein system references previously stored frames to calculate new pose estimates. It increases the memory requirements for the system. Frames above a threshold with minimal distance apart are stored rather than consecutive frames.

The system is not an absolute success and has limitations but is a viable are to explore possibilities of efficient computer based tracking applications for outdoor environments.