Introduction to use Kinect in Unity

Introduction to use Kinect in Unity

1. How to integrate Kinect with Unity in Windows platform.

1. Download following files:

Kinect for Windows SDK

Kinect for Windows Developer Toolkit

Kinect Wrapper package for Unity . This is a sample scene to show how to use Kinect

         in Unity. It contains a scene, some models and some C sharp codes.

2. Install SDK and Developer Toolkit
3. Create a new project in Unity
4. Double click the Kinect Wrapper package for Unity to import it to the new created project.
5. Under Scenes folder, open MainScene
6. Plug in Kinect to computer and Run the project.

2. Brief explanation of sample package

   The sample package includes several C# source code files. We are only interested in the files

   related to gesture recognization. Kinect could recognize 20 joints of human body and provide

   position data of the 20 joints to developer. All the functions are in Kinect10.dll files. After you

   install Kinect SDK, Kinect10.dll will appear in windows/System32 folder. To use the function

   in this dll file, we need to load this dll file first in Unity. You can find codes about this in

   KinectInterop.cs. There is a function called NuiSkeletonGetNextFrame, this is a native   

   method and it provides data of skeleton. This function is called in KinectSensor.cs which save

   skeleton data to NuiSkeletonFrame skeletonFrame object. Skeleton data in this object was

   was used in SkeletonWrapper.cs which made some matrix multiplication. But be honest, we

   do not know what this matrix multiplication are doing. Need time to figure it out. Finally,

   SkeletonWrapper.cs save after-processed data to bonePos array, boneVel array and other

   similar arraies. Developer who need skeleton information just need to call this arraies since

   they are public.

3. Gesture recognization

In our work, we could recognize two basic states of users’ two hands: movement and stationary. Base on the two states, we could recognize more gestures including: Pick, Drop, Discard, Zoom in and Zoom out.

3.1 Move and Stationary

In general, the keyword is the “velocity” of two hands. We could get position data of two hands from Kinect every frame. Then we use  ((currentFramePos - previousFramePos)/deltaTime) to get velocity of two hands at every frame. It seems very simple to recognize state of hands with velocity. If current velocity is greater than 1.0(threshold we set), we consider the hands are moving. If current velocity is less than 1.0, we consider the hands are stationary. During the development process, we found that there always are confusing data appearing. For example, even users think they are keeping their hands stationary, but actually their hands are slightly moving because of shanke which people can not aware of.

To solve this issue, we evaluate the state every 10 frames. If at 8 or more frames, the velocity is greater than the threshold, we consider that users’ hands are moving. Otherwise we think their hands are stationary.

3.2 Pick

Precisely, Pick is not a gesture. If users move their hands in real world, two virtual hands also move in virtual space. If two hands collide with same object, the object will enter controlled state. In this state, users can use gesture to control the object including drop, discard, zoom in and zoom out.

3.3 Drop

In 1.1, we talk out how to recognize movement and stationary state of two hands, If two hands keep stationary, the controlled object will be dropped and users will lost control of the object.

3.4 Discard

If user quickly swipe their two hands back, such as swipe back about shoulder, the object will disappear from the scene. The logic is simple:

1. Set velocity threshold to a big number, such as 800.
2. If velocity of two hands are greater than 800, we know that users are moving two hands

         very quickly. And if velocity value is positive, it means hands are moving back and

         negative value means hands are moving forth. So if users are moving hands back very

         quickly, the velocity should be a value greater than 800.

3. Destroy the controlled object.  

3.5 Zoom in and Zoom out

If users move their left hands to the left(velocity is negative value) and move their hands to the right(velocity is positive value), we consider they are zooming in the controlled object.

Similarly, If users move their left hands to the right(velocity is positive value) and move their hands to the left(velocity is negative value), we consider they are zooming out the controlled object.

4 Walking in Place:

Walking in place refers to making the user to move forward while he just walks by standing there itself. For doing this, the transform coordinates of the player are incremented every frame.

5 Hitting the Player with many cubes:

The player is hit with multiple cubes and on each hit a message is displayed as to where the cube hit the player that is at which joint the cube hit the player. Along with the message, an Image is also displayed.

5.1 Tracking the hand:

The hand of the player in the scene is tracked by a sphere that is the sphere moves in connection with the player’s hand.

5.2 Displaying the message “HI” :

Firstly when the player appears on the scene, a Hi message is displayed. After that it can be made to disappear by letting the player move his hand on his head.