Using Azure Cognitive Services FACE API In UWP App

I have been playing around with Cognitive Services for some time now, as I find them mind blowing APIs that can do a lot of Artificial Intelligence tasks without spending much time building the AI algorithms yourself. Take FACE API for example. Normally, the most common way of face detection/recognition is to use Eigenface classification algorithm. For this, you need to know the basics of AI, such as Regression and Classification, as well as the basics of algorithms, such as SVD, Neural Networks and so on. Even if you use a library such as OpenCV, you still need some knowledge of Artificial Intelligence to make sure you used the correct set of parameters from the library.

Cognitive Services, however, make this demand completely obsolete and put the power of AI truly in the hands of common developers. You don’t need to learn complicated mathematics to use the AI tools anymore. This is good because as AI becomes more and more common in software world, it should also be accessible to the lowest common denominator of developers who might not have upper level degrees in science and mathematics but who know how to code. The downside of it? The APIs are somewhat restricted in what they can do. But this might change in the future (Or so we should hope.)

Today, I am going to talk about using the Face API from Microsoft Azure Cognitive Services, to build a simple UWP application that can tell some of the characteristics of a face (such as age, emotion, smile, facial hair etc.).

To do so, first open Visual Studio –> Create New Project –> Select Windows Universal Blank App. This is your UWP application. To use Azure Cognitive Services in this project, you will have to do two things.

1. Subscribe to Face API in Azure Cognitive Services portal – To do so, go to Cognitive Services Subscription, log on with your account, and select Face API. Once you do it, you will see the following.
   
   
   
   You need the key from here later in the code to subscribe to Face API client.

2. Go To Solution Explorer in Visual Studio –> Right click on Project –> Manage NuGet Packages –> Search for Microsoft.ProjectOxford.Face and install it. Then, in MainPage.xaml.cs, add the following lines of code at the top.
   

   using Microsoft.ProjectOxford.Face;
   using Microsoft.ProjectOxford.Face.Contract;
   using Microsoft.ProjectOxford.Common.Contract;

Now, you are all set to create the Universal App for Face Detection using Cognitive Services.

To do so, you need to do the following things -

1. Access the device camera
2. Run the camera stream in the app
3. Capture the image
4. Call Face API.

We will look into them one by one.

First, let us access the camera and start streaming the video in the app. To do so, in MainPage.xaml, add CaptureElement. See the following code snippet.

<CaptureElement Name="PreviewControl" Stretch="Uniform" Margin="0,0,0,0" Grid.Row="0"/>

Then, create a Windows.Media.Capture.MediaCapture object and initialize it. Add this as a source to CaptureElement and call the StartPreviewAsync() method. Following code snippet might make it clearer.

try {
    m_mediaCapture = new MediaCapture();
    await m_mediaCapture.InitializeAsync();
    m_displayRequest.RequestActive();
    DisplayInformation.AutoRotationPreferences = DisplayOrientations.Landscape;
    PreviewControl.Source = m_mediaCapture;
    await m_mediaCapture.StartPreviewAsync();
    m_isPreviewing = true;
} catch (Exception ex) {
    //Handle Exception
}

This will start video streaming from the camera in your application. Now, to capture the image and to process it, create a button in MainPage.xaml. Add an event handler to this button. In the event handler, call the FaceServiceClient that you should initialize in the initializing code of your app.

FaceServiceClient fClient = new FaceServiceClient("Here your subscription key");

And then, use this InMemoryRandomAccessStream object to capture the photo in JPG encoding; then call DetectAsync from the FaceServiceClient to get the information about faces. 

using(var captureStream = new InMemoryRandomAccessStream()) {
    await m_mediaCapture.CapturePhotoToStreamAsync(ImageEncodingProperties.CreateJpeg(), captureStream);
    captureStream.Seek(0);
    var faces = await fClient.DetectAsync(captureStream.AsStream(), returnFaceLandmarks: true, returnFaceAttributes: new FaceAttributes().GetAll());
};

returnFaceLandmarks and returnFaceAttributes are two important properties that you need to take care of in order to get the full detection information from the API. When returnFaceLandmarks is set to true, you get all the information of location of your face parts, such as Pupils, Nose, and Mouth and so on. The information that comes back looks like the following.

"faceLandmarks": {
    "pupilLeft": {
        "x": 504.4,
        "y": 202.8
    },
    "pupilRight": {
        "x": 607.7,
        "y": 175.9
    },
    "noseTip": {
        "x": 598.5,
        "y": 250.9
    },
    "mouthLeft": {
        "x": 527.7,
        "y": 298.9
    },
    "mouthRight": {
        "x": 626.4,
        "y": 271.5
    },
    "eyebrowLeftOuter": {
        "x": 452.3,
        "y": 191
    },
    "eyebrowLeftInner": {
        "x": 531.4,
        "y": 180.2
    }
}

In faceAttributes properties, you should give the FaceAttributeType that you want to have such as  the following. In my application, I have created a class from which I am returning all of them in a list with a method called GetAll(). 

FaceAttributeType.Age,
FaceAttributeType.Emotion,
FaceAttributeType.FacialHair,
FaceAttributeType.Gender,
FaceAttributeType.Glasses,
FaceAttributeType.HeadPose,
FaceAttributeType.Smile

The result will look somewhere similar to this.

"faceAttributes": {
    "age": 23.8,
    "gender": "female",
    "headPose": {
        "roll": -16.9,
        "yaw": 21.3,
        "pitch": 0
    },
    "smile": 0.826,
    "facialHair": {
        "moustache": 0,
        "beard": 0,
        "sideburns": 0
    },
    "glasses": "ReadingGlasses",
    "emotion": {
        "anger": 0.103,
        "contempt": 0.003,
        "disgust": 0.038,
        "fear": 0.003,
        "happiness": 0.826,
        "neutral": 0.006,
        "sadness": 0.001,
        "surprise": 0.02
    }
}

These are float values between 0 and 1, 1 being maximum and 0 is being least. In my application, I wrote a basic threshold method that can display if I am smiling, angry, happy, etc. in UI, based on these values. The final result looks like this.

It failed to detect my age (because when I tested it, I was using a warm light bulb for lighting. Tip: Lighting matters a lot in age detection from Face API. Use cold lights if you want to look younger). But apart from that, the other information was quite correct. I was indeed smiling, my face was happy, I wasn’t wearing glasses, and have some beard. 

At a first look, Face API looks really interesting. You can do a lot with other endpoints of API such as verification, identification etc. I will try to cover these other functions in next posts.

Until then, keep learning.