Kinect Development Basics On WPF: Using Color Stream, Depth Stream And Infrared Stream

Kinect is quite a fascinating device which has utilities in many fields. A person playing a game like FIFA using just his body motion or a medical situation can be measured using a Kinect. Though the device  has been on the market for quite a time, but still there are a few applications on the Kinect device. So the motive of these tutorial series is to develop an understanding of Kinect development.

Here, in this blog post, we will be covering some basics of Kinect V2, where our main focus is on three type of streams: Color stream, Depth stream and Infrared stream. There is one other stream: Body Stream, which I will cover in my next post.

COLOR STREAM, DEPTH STREAM and INFRARED STREAM

Our Kinect Sensor is just a device which sends only the raw data to our system, now it is upon a developer to use that raw data in whatever way he feels like. But with Kinect SDK 2.0, we get few additional classes to utilize that data. So, the SDK provides us ColorStream class to access the color data from a Kinect Device.

Similarly, to access Infrared Frames coming from Kinect Device we need InfraredStream class which provides us functionality to access infrared frames and accordingly we can perform an operation.

DepthStream class is one special class which is different from the classes discussed above. Using this class, one can access the data of Depth of a person or object standing in front of the Kinect device. This way we can find to how much distance is a visible object from Kinect sensor.

Here, I am building a WPF application in Visual Studion to perform Kinect functionality. To add Kinect dll file, just follow the steps:

Go to the solution explorer of your WPF project -> right click on references -> Add reference -> Assemblies -> Extensions -> Microsoft Kinect.

Inside the project, open MainWindows.xaml.cs or the .cs file of your page and add the namespace:

using Microsoft.Kinect;

Now in the body of the .cs file, add following code to initial the variables to use kinect in the application: 

KinectSensor _sensor;
MultiSourceFrameReader _reader;

After initializing variables, add the following code into the Windows_Loaded method. Here, we will fetch the default Kinect sensor into _sensor, so that, there is no error. Then, after checking whether sensor is there or not, open the sensor: _sensor.Open(). Then, we will define which Frames will the _reader be accepting. In the code below, we added options to access three frame source types: Color, Depth, Infrared which we can use, 

private void window_Loaded(object sender, RoutedEventArgs e)

{

    _sensor = KinectSensor.GetDefault();

    if (_sensor != null)

    {

        _sensor.Open();

        _reader = _sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color | FrameSourceTypes.Depth | FrameSourceTypes.Infrared);

        _reader.MultiSourceFrameArrived + = Reader_MultiSourceFrameArrived;

    }

}

Now, after we have added code for initializing sensor, assigning values of FrameSourceTypes to the reader, we will move to the XAML file or the designer file. Here, we have to add some code for Image control within the page so that, we can get the Stream of those three frames on that control. The code will be: 

<Grid>

    <Image Name="displayStream" />

    <Button Content="Color" Name="ColorStreamButton" Click="ColorStreamButton_Click" Margin="100,915,0,0" />

    <Button Content="Infrared" Name="InfraredStreamButton" Click="InfraredStreamButton_Click" Margin="180,915,0,0" />

    <Button Content="Depth" Name="DepthStreamButton" Click="DepthStreamButton_Click" Margin="260,915,0,0" />

</Grid>

Now in XAML.CS file add the following code:

int _mode = 0;

private void ColorStreamButton_Click(object sender, RoutedEventArgs e)
{
    _mode = 0;
}

private void DepthStreamButton_Click(object sender, RoutedEventArgs e)
{
    _mode = 1;
}

private void InfraredStreamButton_Click(object sender, RoutedEventArgs e)
{
    _mode = 2;
}

void Reader_MultiSourceFrameArrived(object sender, MultiSourceFrameArrivedEventArgs e)

{

    // Get a reference to the multi-frame

    var reference = e.FrameReference.AcquireFrame();

    // Open color frame

    using(var frame = reference.ColorFrameReference.AcquireFrame())
    {

        if (frame != null)
        {

            if (_mode == 0)
            {
                displayStream.Source = frame.ToBitmap();
            }

        }

    }

    // Open depth frame

    using(var frame = reference.DepthFrameReference.AcquireFrame())
    {

        if (frame != null)
        {

            if (_mode == 1)
            {
                displayStream.Source = frame.ToBitmap();
            }

        }

    }

    // Open infrared frame

    using(var frame = reference.InfraredFrameReference.AcquireFrame())
    {

        if (frame != null)
        {

            if (_mode == 2)
            {
                displayStream.Source = frame.ToBitmap();
            }

        }

    }

}

In the above code, I have used '_mode' to toggle between the states of the video stream and comparing its value in the reader function to toggle in the video stream. Moreover, in this application a file named 'extension.cs' is used which is also available as part of nuget package on nuget library. The file provides functionality to convert Kinect raw data into respective streams.

Now, run the program and test with a Kinect V2. :)

I hope with this blog post, the basics of Kinect development on WPF are covered up. Many more are in the line for Kinect development. Thanks!

Read more articles on WPF:

• Simple WPF TreeView Control Programmatically
• Create A Rich Text Editor Using WPF