dSensingNI - Depth Sensing Natural Interaction

dSensingNI is a new detection and tracking framework for multitouch, freehand and advanced tangible interaction using a depth camera.

dSensingNI was primarily designed for use with tabletops but is also applicable for vertical installations like interactive walls, white boards etc.

To transfer the tracking data between dSensingNI and a client application the TUIO protocol is used. The use of TUIO in combination with dSensingNI allows the rapid development of multitouch and tangible interfaces.

dSensingNI Features

How does it work?

The functionality is simple and effective:

A depth camera (e.g. Microsoft Kinect) is mounted above the interaction area. A demo setup can be seen in the figure on the left.

The depth camera is connected to a computer system, which runs the dSensingNI.

dSensingNI analyzes the depth image based on a number of user-defined parameters and transmits all detected and tracked features (e.g. finger touches, palms, fingers, tangibles) to any client application via TUIO.

dSensingNI - Functionality

On the client side using the TUIO3DLib all detected features are received and passed in a suitable form to the client application components.

The client application handles these features to implement various interaction concepts.

The properties of the tracked features

Below are properties listed, which are recognized by the dSensingNI and transmitted to the client application.

General properties of all features

  • ID - unique identifier for each tracked feature

  • Position - 3D position of the feature (in relative coordinates or in [mm])

  • Distance to the surface - approximate distance (in relative coordinates or in [mm]) of the feature to the user-defined interaction area

  • Velocity - movement speed of the feature

Hand (palm)

  • Pointing direction - pointing direction vector of an arm

Finger

  • Hand ID - assignment to the respective hand

Touch (blob)

  • Touched tangible ID - the possibly touched object (tangible)

Object (tangible)

  • Rotation angle - rotation angle of each tangible

  • Dimension - height, width and depth of each tangible (in relative coordinates or in [mm])

  • Volumen - the volume of the tangible (in relative coordinates or in [mm³])

  • Combined object ID - tangible, with wich this tangible is combined

  • Grasped hand ID - the hand, that grasps this tangible

The TUIO3DLib

Client application developers can use the provided .NET library (TUIO3DLib) to receive all detected and tracked features (e.g. finger touches, palms, fingers, tangibles). It consists of a TUIO3DClient that decodes extended TUIO data and passes this data via predefined events like AddHandEvent, UpdateHandEvent or RemoveHandEvent (see class diagram below) to the client application components.

TUIO3DLib class diagram

Publications

F. Klompmaker, K. Nebe and A. Fast, "dSensingNI – A Framework for Advanced Tangible Interaction using a Depth Camera", In Proc.: Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI), 2012.