Visual Applications and Arduino

My previous articles all dealt with control of Arduino and their control using a WPF app and Windows Phone. Let's scale up things a bit and let's bring image processing to the stage. When we deal with the human-computer interface, we like to deal with easier ways to interface with our digital counterpart or let's say a system. The article will explore the use of computer vision on an Arduino board.

 

Software required

1. Microsoft Visual Studio
2. Arduino IDE 

Hardware required

1. Arduino board
2. A webcam
3. A LED

Libraries used

1. Aforge.NET http://www.aforgenet.com/ 

Introduction

The system that will be discussed here is very simple. What we will do here is build an application in WPF that does some image processing. Then we will use the result of the image processing application to control an Arduino. In this article, we will walk through the concepts of image processing using the Afoge.NET library developed by Andrew Krillov. Using the library, we will develop an application and then go through the same old story of controlling an Arduino by a WPF application. However, the same technique can be applied to a Windows Forms application as well. Here we have used Aforge.NET. We can use the RealSense SDK as well. In fact, RealSense gives more accurate results. I don't own a RealSense device so I will stick to a normal webcam and use Afotge.NET to do some image processing.

Our first course of action is to open Visual Studio and create a new WPF application project. Once we have done that, from the toolbar, we will add the following controls required. 

1. A pictureBox
2. A button for starting the image control
3. A button for stopping the image control
4. A button for connecting to the Arduino
5. A button for disconnecting from the Arduino
6. A listBox for displaying the camera devices available. 

Note: This article won't go into depth about the details of the UI design. Just a screenshot with some brief details are provided. In-depth details will make this lengthy.

 

 

Figure 1: Screenshot of the UI (MainWindow.xaml) of the app.

In the preceding figure we have the components for:

1. Displaying the picture
2. Displaying the status for connection
3. Buttons for activating the app and connecting to the Arduino
4. Buttons 

Approach

Our target is to develop an application that will detectmotions and if motion is detected, then a signal will be sent to the Arduino that will light up a LED. We will use the Aforge.NET library to detect the motion and then use the Serial class to communicate with the Arduino. Since I am dealing with motion detection, I have used some tutorials by Andrew Krillov in codeproject.org. All links are mentioned below under in the "Further studies" section.

Our second step is to code for MainPage.xaml.cs. Before moving on the main code, you need to add the references for using Aforge.NET. I have used the Nuget service manager to add the library. Add these:

1. Aforge.Imaging
2. Aforge.Math
3. Aforge.Video
4. Aforge.Video.DirectShow
5. Aforge.Vision

Now, after adding these, include the following in your namespaces:

using System;
using System.Collections.Generic;
using System.Linq;
using System.ComponentModel;
using System.Text;
using System.Drawing;
using System.Threading.Tasks;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
-using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Navigation;
using System.Windows.Shapes;
using AForge.Video;
using AForge;
using AForge.Math.Geometry;
using AForge.Video.DirectShow;
using AForge.Imaging;
using AForge.Imaging.Filters;
using System.IO;
using System.Drawing.Imaging;
using System.Threading;
using System.IO.Ports;

Note: Do not forget to create the event handlers of the components in MainPage.xaml.

 

Now add the Serial Port class. 

SerialPort sp = new SerialPort();

Initialize the global variables. 

int height, width;
//Initialize the background frame to null
Bitmap backgroundframe = null;
private FilterInfoCollection videoDevices;
private VideoCaptureDevice videoSource = null;

Create a method and name it getCamList() that will be used for getting the camera list available in our device. The cam list will be displayed in the combobox. 

private void getCamList()
{

   videoDevices = new FilterInfoCollection(FilterCategory.VideoInputDevice);
   foreach (FilterInfo device in videoDevices)
   {
       //cambox is the name of the combobox used in the UI
       cambox.Items.Add(device.Name);
   }
   cambox.SelectedIndex = 0; //make dafault to first cam
}

In the preceding method, we added the available cameras to our combobox and then by default we have selected index 0 since here I have assumed that the 0 index device is your primary device. Now go to MainPage.xaml and create the window loaded event. In the event handler method, we will call the preceding method.

private void Window_Loaded(object sender, RoutedEventArgs e)
{
     getCamList();
}

Now we will write code for the click event for the Start button.

private void start_Click(object sender, RoutedEventArgs e)
{
      stop.IsEnabled = true;
      start.IsEnabled = false;
      videoSource = new VideoCaptureDevice(videoDevices[cambox.SelectedIndex].MonikerString);
      videoSource.NewFrame += new NewFrameEventHandler(video_NewFrame);
      //videoSource.DesiredFrameRate = 10;
      videoSource.Start();
}

In the preceding method, most of the work is starightforward and we have created a new event for getting a new frame. Also we started the video source and in our further processing part, we will use method videoNewFrame. Now browse to the video newframe method and add these lines initially.

//Input image
Bitmap IPimage = (Bitmap)eventArgs.Frame.Clone();
//Intialize the current frame
Bitmap currentframe = IPimage;

In the preceding code, we created a Bitmap named IPimage that is a clone of the input frame. Then we took a new Bitmap currentframe and then initialized it to the input frame. Our next job is to create the filters that will be applied to the input image.

Grayscale filter = new Grayscale(0.2125, 0.7154, 0.0721);
//Create difference filter
Difference differenceFilter = new Difference();
//create threshold filter
IFilter thresholdFilter = new Threshold(15);

In the preceding lines, we created the Grayscale filter, the difference filter and the threshold filter. These filters will be used for the motion detection. The approach is to have two frames. The first is the background frame and the next will be the current frame. By using the difference and the threshold filter, we get the newly added areas that signify motion.

Now for comparing the current frame with the background frame, write this code.

int framesize;
width = IPimage.Width;
height = IPimage.Height;
//Calculate the frame size
framesize = width * height;
if (backgroundframe == null)
{
    BitmapData bitmapData = IPimage.LockBits(
    new System.Drawing.Rectangle(0, 0, width, height),
    ImageLockMode.ReadOnly, System.Drawing.Imaging.PixelFormat.Format24bppRgb);
    // apply grayscale filter getting unmanaged image
    backgroundframe = filter.Apply(new UnmanagedImage(bitmapData)).ToManagedImage();
    // unlock source image
    IPimage.UnlockBits(bitmapData);
}
currentframe = filter.Apply(currentframe);
differenceFilter.OverlayImage = backgroundframe;
// apply the filters
Bitmap tmp1 = differenceFilter.Apply(currentframe);
Bitmap tmp2 = thresholdFilter.Apply(tmp1);

In the preceding code, we first got the background frame and then locked the source image. After getting the background frame we unlock the source frame and then initialize the current frame to the input frame. Then the difference filter and the threshold filter is applied to get our job done.

Noise is a factor when we deal with sensors. Similarly we need filters to reduce noise to an optimum level. Here the erosion filter is used to do that.

IFilter erosionFilter = new Erosion();
// apply the filter
Bitmap tmp3 = erosionFilter.Apply(tmp2);

Now we have a detector ready but it's of no use unless we highlight them. Let's include this BlobCounter class for our job. 

     BlobCounter blobCounter = new BlobCounter();
    //get object rectangles
    blobCounter.ProcessImage(tmp2);
    System.Drawing.Rectangle[] rects = blobCounter.GetObjectsRectangles();
    // create graphics object from initial image
    Graphics g = Graphics.FromImage(IPimage);
    // draw each rectangle
    using (System.Drawing.Pen pen = new System.Drawing.Pen(System.Drawing.Color.Red, 1))
    {
        foreach (System.Drawing.Rectangle rc in rects)
        {


            if ((rc.Width > 50) && (rc.Height > 50) && (rc.Width < 200) && (rc.Height < 200))

            {
                g.DrawRectangle(pen, rc);
                //Send a serial character to the Arduino for some alarm or for some operation
                sp.Write("1");
            }
        }
    }
    g.Dispose();
    MemoryStream ms = new MemoryStream();
    IPimage.Save(ms, ImageFormat.Bmp);
    ms.Seek(0, SeekOrigin.Begin);
    BitmapImage bi = new BitmapImage();
    bi.BeginInit();
    bi.StreamSource = ms;
    bi.EndInit();

    bi.Freeze();
    Dispatcher.BeginInvoke(new ThreadStart(delegate
    {
        imbox.Source = bi;
    }));
}

By using the Blob counter class, we get the motioned area's length and breadth of the number of motions and so on. In the preceding code, I have included a condition that if the width and height lie between 50 and 200, we will highlight that area and also at the same time send a serial character to the Arduino. That's it, your application for detection of motion is ready and just plug in your Arduino and connect some jumpers to see the magic.

The rest of the code is simple. Just add the code for connecting to the Arduino using the eventhandler methods for the buttons. Now if you are not able to understand the connection method, just have a look at my previous article where I described the development of a WPF application for an Arduino control.

private void stop_Click(object sender, RoutedEventArgs e)
{
    videoSource.Stop();
}

private void Connect_Click(object sender, RoutedEventArgs e)
{
    try
    {
        String portName = comno.Text;
        sp.PortName = portName;
        sp.BaudRate = 9600;
        sp.Open();
        //s1.Text = "Connected";
    }
    catch (Exception)
    {

        MessageBox.Show("Please give a valid port number or check your connection");
    }
}

private void Disconnect_Click(object sender, RoutedEventArgs e)
{
    sp.Close();
}

Congratsulations! Your application is ready.

I have included the project with this article. You may download it and use it, no issues with that. Let's move to the Arduino part. Well, in my previous articles, I already wrote about serial communication in Arduino. Have a look at my previous article. The circuitry for the Arduino is simple. You just need to have a LED attached to pin 13 (the long lead of the LED is connected to pin 13 and the shorter one to the Gnd pin on the Arduino) of the Arduino. Plug in your Arduino with the USB and burn in the following code.

void setup() {
  // put your setup code here, to run once:
  pinMode(13, OUTPUT);
  Serial.begin(9600);

}

void loop() {
  // put your main code here, to run repeatedly:
 char c= Serial.read();
 if(c=='1')
  digitalWrite(13,HIGH);
else
  digitalWrite(13,LOW);
}

In the preceding Arduino code, we have read a character using the serial port and if it is equal to '1', then we just change pin 13 to high that switches on a LED.

The image below provides an idea of the output to be displayed. The Red boxes are the blobs and using the BlobCounter class, it is possible to get the number of blobs, the size and the height.

 

 

Further study

1: Motion detection Algorithm by Andrew Krillov.

2: Arduino control using a WPF application

 

Summary

In this article, we saw how image processing can be used in a WPF application and how to combine image processing with an Arduino and develop a very basic version of a motion detector. Now, you can develop other computer vision applications with Arduino and also implement it in robotics. In case of any problems, comment below or contact me by e-mail.