Introduction
Microsoft Cognitive Services (earlier known as Project Oxford) provide us the ability to build intelligent applications just by writing a few lines of code. These applications or services are deployed on all major platforms like Windows, iOS, and Android. All the APIs are based on machine learning APIs and enable the developers to easily add intelligent features – such as emotion and video detection; facial, speech and vision recognition; and speech and language understanding – into their applications.
Look into the below URLs for reference of Microsoft Cognitive Services.
- https://azure.microsoft.com/en-in/services/cognitive-services/
- https://docs.microsoft.com/en-us/azure/cognitive-services/welcome
NOTE
Microsoft announced the preview version of Microsoft Cognitive Services on March 30, 2016.
I would recommend you to read the articles mentioned below.
In this article, I will walk you through steps of exploring FACE and EMOTION APIs of Microsoft Cognitive Services. Henceforth, this article will cover the following three parts and I would suggest you go through this article in the below-given order only.
- Create Azure Account
- FACE API
- EMOTION API
So, below are the prerequisites to work with Microsoft Cognitive Services API,
- Visual Studio 2015 (Community, Enterprise, or Professional edition)
- Microsoft Azure Account
Now, let’s get started.
Part 1 – Create Azure Account
- Sign in to the Microsoft Azure Portal.
- You will be asked to log in with a Microsoft Account. You can take a free subscription of one month or can choose among different plans available on the Azure portal, as per your requirement or business needs. (In my case, I took free subscription of one month).
- You will be asked for your phone number and credit card details.
- You will be given some credit points after successful creation of your account based on your selected country and zone.
- After successful creation of Azure Account, you will see the dashboard as shown below.
Part 2 – FACE API
Face API, provided by Microsoft Cognitive Services, helps to detect, analyze, and organize the faces in a given image. We can also tag faces in any given photo.
Face API provides the advanced face algorithms and it has two main functions:-
- Face Detection with Attributes
- Face API detects up to 64 human faces with high precision face location in an image.
- The image can be specified by file in bytes or valid URL.
- Face Recognition
- Face Verification
- Performs an authentication against two detected faces or authentication from one detected face to one-person object.
- Finding Similar Face
- It takes target face or query face as input and finds a small set of faces that looks most similar to the target face. It has two modes – matchFace and matchPerson.
- matchFace returns similar faces, as it ignores the same-person threshold
- matchPerson returns similar faces after applying a same-person threshold
- Face Grouping
- It takes a set of unknown faces as input
- Face Grouping API divides these unknown faces into several groups based on similarity.
- Personal (Face) Identification
- It identifies people based on a detected face and people database. This database needs to be created in advance and can be edited later.
Assuming you have Azure portal account, follow the below steps to implement FACE API.
Step 1
Click on “+” or “New” link in the Azure portal on left-hand side.
Step 2
Once you click on “AI + Cognitive Services”, you will see the list of APIs available in Cognitive Services.
Step 3
Choose Face API to subscribe to Microsoft Cognitive Services Face API and proceed further with subscription steps. After clicking on Face API, it will show the legal page. Read it carefully and then click on "Create".
Step 4
After clicking on Create, there are two possibilities (happened in my case).
- You will be given a form to fill, as shown below, in which you have to fill the below details (This option might be visible after a few hours)
- Name
- Subscription (In my case, I choose Free trial; you can choose F0 or S0 subscription types)
- Resource Group
- You will see the below image and it will ask you for a new subscription, in which case you can generate the Subscription Keys and Endpoint URL from a different URL (https://azure.microsoft.com/en-us/try/cognitive-services/).
Below are the generated Keys and Endpoint URL.
Step 5
Create a WPF Application in Visual Studio 2015 (Visual C# > Windows Desktop > WPF Application). I have named the application as “Face Tutorial”.
Step 6
Add a button with the name as “Browse” on MainWindow.xaml, using designer or code. Here, I prefer adding the button using code.
- <Window x:Class="Face_Tutorial.MainWindow"
- xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
- xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
- Title="MainWindow" Height="700" Width="960">
- <Grid x:Name="BackPanel">
- <Image x:Name="FacePhoto" Stretch="Uniform" Margin="0,0,0,50" MouseMove="FacePhoto_MouseMove" />
- <DockPanel DockPanel.Dock="Bottom">
- <Button x:Name="BrowseButton" Width="72" Height="20" VerticalAlignment="Bottom" HorizontalAlignment="Left"
- Content="Browse..."
- Click="BrowseButton_Click" />
- <StatusBar VerticalAlignment="Bottom">
- <StatusBarItem>
- <TextBlock Name="faceDescriptionStatusBar" />
- </StatusBarItem>
- </StatusBar>
- </DockPanel>
- </Grid>
- </Window>
Step 7
Now, go to MainWindow.xaml.cs. The below-given directives are required in the solution to access the Face APIs.
- using Microsoft.ProjectOxford.Face;
- using Microsoft.ProjectOxford.Face.Contract;
To add the above references, browse for two dll’s and click on "Install".
- Newtonsoft.JSON
- Microsoft.ProjectOxford.Face
Once you add these two dll’s, these will be shown in the solution.
Step 8
Add the lines of code given below to click event of Browse button.
- private async void BrowseButton_Click(object sender, RoutedEventArgs e)
- {
-
- var openDlg = new Microsoft.Win32.OpenFileDialog();
-
- openDlg.Filter = "JPEG Image(*.jpg)|*.jpg";
- bool? result = openDlg.ShowDialog(this);
-
-
- if (!(bool)result)
- {
- return;
- }
-
-
- string filePath = openDlg.FileName;
-
- Uri fileUri = new Uri(filePath);
- BitmapImage bitmapSource = new BitmapImage();
-
- bitmapSource.BeginInit();
- bitmapSource.CacheOption = BitmapCacheOption.None;
- bitmapSource.UriSource = fileUri;
- bitmapSource.EndInit();
-
- FacePhoto.Source = bitmapSource;
-
-
- Title = "Detecting...";
- faces = await UploadAndDetectFaces(filePath);
- Title = String.Format("Detection Finished. {0} face(s) detected", faces.Length);
-
- if(faces.Length > 0)
- {
-
- DrawingVisual visual = new DrawingVisual();
- DrawingContext drawingContext = visual.RenderOpen();
- drawingContext.DrawImage(bitmapSource, new Rect(0, 0, bitmapSource.Width, bitmapSource.Height));
- double dpi = bitmapSource.DpiX;
- resizeFactor = 96 / dpi;
- faceDescriptions = new String[faces.Length];
-
- for (int i = 0; i < faces.Length; ++i)
- {
- Face face = faces[i];
-
-
- drawingContext.DrawRectangle(
- Brushes.Transparent,
- new Pen(Brushes.Red, 2),
- new Rect(
- face.FaceRectangle.Left * resizeFactor,
- face.FaceRectangle.Top * resizeFactor,
- face.FaceRectangle.Width * resizeFactor,
- face.FaceRectangle.Height * resizeFactor
- )
- );
-
-
- faceDescriptions[i] = FaceDescription(face);
- }
-
- drawingContext.Close();
-
-
- RenderTargetBitmap faceWithRectBitmap = new RenderTargetBitmap(
- (int)(bitmapSource.PixelWidth * resizeFactor),
- (int)(bitmapSource.PixelHeight * resizeFactor),
- 96,
- 96,
- PixelFormats.Pbgra32);
-
- faceWithRectBitmap.Render(visual);
- FacePhoto.Source = faceWithRectBitmap;
-
-
- faceDescriptionStatusBar.Text = "Place the mouse pointer over a face to see the face description.";
- }
- }
Add a new await function named UploadAndDetectFaces(), which accepts imageFilePath as an object parameter.
-
- private async Task<Face[]> UploadAndDetectFaces(string imageFilePath)
- {
-
- IEnumerable<FaceAttributeType> faceAttributes =
- new FaceAttributeType[] { FaceAttributeType.Gender, FaceAttributeType.Age, FaceAttributeType.Smile, FaceAttributeType.Emotion, FaceAttributeType.Glasses, FaceAttributeType.Hair };
-
-
- try
- {
- using (Stream imageFileStream = File.OpenRead(imageFilePath))
- {
- Face[] faces = await faceServiceClient.DetectAsync(imageFileStream, returnFaceId: true, returnFaceLandmarks: false, returnFaceAttributes: faceAttributes);
- return faces;
- }
- }
-
- catch (FaceAPIException f)
- {
- MessageBox.Show(f.ErrorMessage, f.ErrorCode);
- return new Face[0];
- }
-
- catch (Exception e)
- {
- MessageBox.Show(e.Message, "Error");
- return new Face[0];
- }
- }
Below is the code which appends the string as output in the status bar.
-
- private string FaceDescription(Face face)
- {
- StringBuilder sb = new StringBuilder();
-
- sb.Append("Face: ");
-
-
- sb.Append(face.FaceAttributes.Gender);
- sb.Append(", ");
- sb.Append(face.FaceAttributes.Age);
- sb.Append(", ");
- sb.Append(String.Format("smile {0:F1}%, ", face.FaceAttributes.Smile * 100));
-
-
- sb.Append("Emotion: ");
- EmotionScores emotionScores = face.FaceAttributes.Emotion;
- if (emotionScores.Anger >= 0.1f) sb.Append(String.Format("anger {0:F1}%, ", emotionScores.Anger * 100));
- if (emotionScores.Contempt >= 0.1f) sb.Append(String.Format("contempt {0:F1}%, ", emotionScores.Contempt * 100));
- if (emotionScores.Disgust >= 0.1f) sb.Append(String.Format("disgust {0:F1}%, ", emotionScores.Disgust * 100));
- if (emotionScores.Fear >= 0.1f) sb.Append(String.Format("fear {0:F1}%, ", emotionScores.Fear * 100));
- if (emotionScores.Happiness >= 0.1f) sb.Append(String.Format("happiness {0:F1}%, ", emotionScores.Happiness * 100));
- if (emotionScores.Neutral >= 0.1f) sb.Append(String.Format("neutral {0:F1}%, ", emotionScores.Neutral * 100));
- if (emotionScores.Sadness >= 0.1f) sb.Append(String.Format("sadness {0:F1}%, ", emotionScores.Sadness * 100));
- if (emotionScores.Surprise >= 0.1f) sb.Append(String.Format("surprise {0:F1}%, ", emotionScores.Surprise * 100));
-
-
- sb.Append(face.FaceAttributes.Glasses);
- sb.Append(", ");
-
-
- sb.Append("Hair: ");
-
-
- if (face.FaceAttributes.Hair.Bald >= 0.01f)
- sb.Append(String.Format("bald {0:F1}% ", face.FaceAttributes.Hair.Bald * 100));
-
-
- HairColor[] hairColors = face.FaceAttributes.Hair.HairColor;
- foreach (HairColor hairColor in hairColors)
- {
- if (hairColor.Confidence >= 0.1f)
- {
- sb.Append(hairColor.Color.ToString());
- sb.Append(String.Format(" {0:F1}% ", hairColor.Confidence * 100));
- }
- }
-
-
- return sb.ToString();
- }
Finally, add a function, which returns the string on mouse hover over the image.
- private void FacePhoto_MouseMove(object sender, MouseEventArgs e)
- {
-
- if (faces == null)
- return;
-
-
- Point mouseXY = e.GetPosition(FacePhoto);
-
- ImageSource imageSource = FacePhoto.Source;
- BitmapSource bitmapSource = (BitmapSource)imageSource;
-
-
- var scale = FacePhoto.ActualWidth / (bitmapSource.PixelWidth / resizeFactor);
-
-
- bool mouseOverFace = false;
-
- for (int i = 0; i < faces.Length; ++i)
- {
- FaceRectangle fr = faces[i].FaceRectangle;
- double left = fr.Left * scale;
- double top = fr.Top * scale;
- double width = fr.Width * scale;
- double height = fr.Height * scale;
-
-
- if (mouseXY.X >= left && mouseXY.X <= left + width && mouseXY.Y >= top && mouseXY.Y <= top + height)
- {
- faceDescriptionStatusBar.Text = faceDescriptions[i];
- mouseOverFace = true;
- break;
- }
- }
-
-
- if (!mouseOverFace)
- faceDescriptionStatusBar.Text = "Place the mouse pointer over a face to see the face description.";
- }
Step 9
Build the application and run it. You will see the output as below.
Summary
Therefore, in this part, we saw that by writing very less code, we can use Microsoft Cognitive Services Face API.
Part 3 – EMOTION API
Emotion API provides the advanced Emotion algorithms and it has two main functions:-
- Emotion Recognition
- It takes image as an input and returns the confidence of the emotions for each face in the image.
- The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.
- Emotion scores are normalized to sum to one.
- Emotion in Video
- It takes video as an input and returns the confidence across a set of emotions for the group of faces in the image over a period.
- The emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust or neutral.
- It returns two types of aggregates,
- windowMeanScores gives a mean score for all of the faces detected in a frame for each emotion.
- windowFaceDistribution gives the distribution of faces with each emotion as the dominant emotion for that face.
Now, let’s create a C# Console application to test Microsoft Cognitive Services Emotion API.
Step 1
We have already created the Endpoint URL and subscription keys in Azure for Emotion API. We will create a Console Application in Visual Studio 2015.
Step 2
Replace the Program.cs with the following code.
- using System;
- using System.Collections.Generic;
- using System.Linq;
- using System.Text;
- using System.Threading.Tasks;
- using System.IO;
- using System.Net.Http;
- using System.Net.Http.Headers;
-
- namespace Emotion_API_Tutorial
- {
- class Program
- {
- static void Main(string[] args)
- {
- Console.WriteLine("Enter the path to a JPEG image file:");
- string imageFilePath = Console.ReadLine();
-
- MakeRequest(imageFilePath);
-
- Console.WriteLine("\n\n\nWait for the result below, then hit ENTER to exit...\n\n\n");
- Console.ReadLine();
- }
-
- static byte[] GetImageAsByteArray(string imageFilePath)
- {
- FileStream fileStream = new FileStream(imageFilePath, FileMode.Open, FileAccess.Read);
- BinaryReader binaryReader = new BinaryReader(fileStream);
- return binaryReader.ReadBytes((int)fileStream.Length);
- }
-
- static async void MakeRequest(string imageFilePath)
- {
- try
- {
- var client = new HttpClient();
-
- client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "<Enter Your Key Value>");
-
-
- string uri = "https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize?";
-
- HttpResponseMessage response;
- string responseContent;
-
-
- byte[] byteData = GetImageAsByteArray(imageFilePath);
-
- using (var content = new ByteArrayContent(byteData))
- {
-
-
- content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
- response = await client.PostAsync(uri, content);
- responseContent = response.Content.ReadAsStringAsync().Result;
- }
-
-
- Console.WriteLine(responseContent);
- }
- catch (Exception ex)
- {
- Console.WriteLine("Error occured:= " + ex.Message + ex.StackTrace);
- Console.ReadLine();
- }
- }
- }
- }
Step 3
Build and run the application. A successful call will return an array of face entries and their emotion scores. An empty response indicates that no faces were detected. An emotion entry contains the following fields.
- faceRectangle - Rectangle location of face in the image.
- scores - Emotion scores for each face in the image.
Below is the output of my trial run. Provide the path of an image from your local folder and hit Enter.
- [
- {
- "faceRectangle": {
- "height": 44,
- "left": 62,
- "top": 36,
- "width": 44
- },
- "scores": {
- "anger": 0.000009850864,
- "contempt": 1.073325e-8,
- "disgust": 0.00000230705427,
- "fear": 1.63113334e-9,
- "happiness": 0.9999875,
- "neutral": 1.00619431e-7,
- "sadness": 1.13927945e-9,
- "surprise": 2.365794e-7
- }
- }
- ]
Summary
Therefore, in this part, we saw that by writing much less code, we could use Microsoft Cognitive Services Emotion API.