dSensingNI is a new detection and tracking framework for multitouch, freehand and advanced tangible interaction using a depth camera.
dSensingNI was primarily designed for use with tabletops but is also applicable for vertical installations like interactive walls, white boards etc.
To transfer the tracking data between dSensingNI and a client application the TUIO protocol is used. The use of TUIO in combination with dSensingNI allows the rapid development of multitouch and tangible interfaces.
- Tracking of finger touches on arbitrary non-planar surfaces and even on the surface of any movable object (multitouch interaction)
- Any arbitrary object can easily be used for interaction (tangible interaction) - no integrated logic or electrical circuits are needed
- Detection of arms, palm and fingers above and/or in front of the interaction area (freehand interaction)
- Detection of object-object interaction (e.g. object stacking or grouping) as well as object-hand interactions (e.g. grasping, moving and releasing)
- Interaction in 3D space
- Transmission of an abstract description of all detected and tracked
features to any client application via TUIO
(extended TUIO 1.1 protocol)
- Rapid development of client applications with the use of the provided client library (TUIO3DLib)
- Reuse of existing constructions or large-area displays to extend them with multi-touch, freehand and tangible ability
- Combination of multitouch, freehand and advanced tangible interaction
How does it work?
The functionality is simple and effective:
A depth camera (e.g. Microsoft Kinect) is mounted above the interaction area. A demo setup can be seen in the figure on the left.
The depth camera is connected to a computer system, which runs the dSensingNI.
dSensingNI analyzes the depth image based on a number of user-defined parameters and transmits all detected and tracked features (e.g. finger touches, palms, fingers, tangibles) to any client application via TUIO.
On the client side using the TUIO3DLib all detected features are received and passed in a suitable form to the client application components.
The client application handles these features to implement various interaction concepts.
The properties of the tracked features
Below are properties listed, which are recognized by the dSensingNI and transmitted to the client application.
General properties of all features
ID - unique identifier for each tracked feature
Position - 3D position of the feature (in relative coordinates or in [mm])
Distance to the surface - approximate distance (in relative coordinates or in [mm]) of the feature to the user-defined interaction area
Velocity - movement speed of the feature
Pointing direction - pointing direction vector of an arm
Hand ID - assignment to the respective hand
Touched tangible ID - the possibly touched object (tangible)
Rotation angle - rotation angle of each tangible
Dimension - height, width and depth of each tangible (in relative coordinates or in [mm])
Volumen - the volume of the tangible (in relative coordinates or in [mm³])
Combined object ID - tangible, with wich this tangible is combined
Grasped hand ID - the hand, that grasps this tangible
Client application developers can use the provided .NET library (TUIO3DLib) to receive all detected and tracked features (e.g. finger touches, palms, fingers, tangibles). It consists of a TUIO3DClient that decodes extended TUIO data and passes this data via predefined events like AddHandEvent, UpdateHandEvent or RemoveHandEvent (see class diagram below) to the client application components.
F. Klompmaker, K. Nebe and A. Fast, "dSensingNI – A Framework for Advanced Tangible Interaction using a Depth Camera", In Proc.: Sixth International Conference on Tangible, Embedded and Embodied Interaction (TEI), 2012.