Introduction to use Kinect in Unity
1. How to integrate Kinect with Unity in Windows platform.
Download following files:
in Unity. It contains a scene, some models and some C sharp codes.
Install SDK and Developer Toolkit
Create a new project in Unity
-
Under Scenes folder, open MainScene
Plug in Kinect to computer and Run the project.
2. Brief explanation of sample package
The sample package includes several C# source code files. We are only interested in the files
related to gesture recognization. Kinect could recognize 20 joints of human body and provide
position data of the 20 joints to developer. All the functions are in Kinect10.dll files. After you
install Kinect SDK, Kinect10.dll will appear in windows/System32 folder. To use the function
in this dll file, we need to load this dll file first in Unity. You can find codes about this in
KinectInterop.cs. There is a function called NuiSkeletonGetNextFrame, this is a native
method and it provides data of skeleton. This function is called in KinectSensor.cs which save
skeleton data to NuiSkeletonFrame skeletonFrame object. Skeleton data in this object was
was used in SkeletonWrapper.cs which made some matrix multiplication. But be honest, we
do not know what this matrix multiplication are doing. Need time to figure it out. Finally,
SkeletonWrapper.cs save after-processed data to bonePos array, boneVel array and other
similar arraies. Developer who need skeleton information just need to call this arraies since
they are public.
3. Gesture recognization
In our work, we could recognize two basic states of users’ two hands: movement and stationary. Base on the two states, we could recognize more gestures including: Pick, Drop, Discard, Zoom in and Zoom out.
3.1 Move and Stationary
In general, the keyword is the “velocity” of two hands. We could get position data of two hands from Kinect every frame. Then we use ((currentFramePos - previousFramePos)/deltaTime) to get velocity of two hands at every frame. It seems very simple to recognize state of hands with velocity. If current velocity is greater than 1.0(threshold we set), we consider the hands are moving. If current velocity is less than 1.0, we consider the hands are stationary. During the development process, we found that there always are confusing data appearing. For example, even users think they are keeping their hands stationary, but actually their hands are slightly moving because of shanke which people can not aware of.
To solve this issue, we evaluate the state every 10 frames. If at 8 or more frames, the velocity is greater than the threshold, we consider that users’ hands are moving. Otherwise we think their hands are stationary.
3.2 Pick
Precisely, Pick is not a gesture. If users move their hands in real world, two virtual hands also move in virtual space. If two hands collide with same object, the object will enter controlled state. In this state, users can use gesture to control the object including drop, discard, zoom in and zoom out.
3.3 Drop
In 1.1, we talk out how to recognize movement and stationary state of two hands, If two hands keep stationary, the controlled object will be dropped and users will lost control of the object.
3.4 Discard
If user quickly swipe their two hands back, such as swipe back about shoulder, the object will disappear from the scene. The logic is simple:
Set velocity threshold to a big number, such as 800.
If velocity of two hands are greater than 800, we know that users are moving two hands
very quickly. And if velocity value is positive, it means hands are moving back and
negative value means hands are moving forth. So if users are moving hands back very
quickly, the velocity should be a value greater than 800.
Destroy the controlled object.
3.5 Zoom in and Zoom out
If users move their left hands to the left(velocity is negative value) and move their hands to the right(velocity is positive value), we consider they are zooming in the controlled object.
Similarly, If users move their left hands to the right(velocity is positive value) and move their hands to the left(velocity is negative value), we consider they are zooming out the controlled object.
4 Walking in Place:
Walking in place refers to making the user to move forward while he just walks by standing there itself. For doing this, the transform coordinates of the player are incremented every frame.
5 Hitting the Player with many cubes:
The player is hit with multiple cubes and on each hit a message is displayed as to where the cube hit the player that is at which joint the cube hit the player. Along with the message, an Image is also displayed.
5.1 Tracking the hand:
The hand of the player in the scene is tracked by a sphere that is the sphere moves in connection with the player’s hand.
5.2 Displaying the message “HI” :
Firstly when the player appears on the scene, a Hi message is displayed. After that it can be made to disappear by letting the player move his hand on his head.