Top Five Pieces of Advice For Users Of Microsoft Cognitive Services

Introduction and Background

As the title suggests, this post is a personal recommendation for the users of Microsoft Cognitive Services, as these are the Services, which provide a Cloud-based subscription-based solution for artificially intelligent software applications, within any team, any purpose and any scale commitment. We all know Microsoft is investing a lot in manpower, promotion and commitment in Azure nowadays and almost every solution is dancing around the margins  of Azure in one way or the other. They come back to the same conclusion, that the solution can be purchased as a software-as-a-Service from Azure — there are many other names, Platform-as-a-Service, Service-as-a-Service, choose yours from the pool, as liked.

In this post, I am going to cover up the most important points that your team should understand before migrating to Microsoft Cognitive Services.

Background of Microsoft Cognitive Services

Anyone, who has no idea of what Microsoft Cognitive Services are Microsoft Cognitive Services are a bundle of Services, which are provided by Microsoft to the individuals, team and/or organizations of any size and any scale to provide Services that require complex machine learning or artificially intelligent responses.

It is a tough task to accomplish the machine learning and with just one wrong input, your entire algorithm can go to {add slang here}. Microsoft is providing the Service, where you must only provide the inputs for the algorithm and you get the output. Microsoft itself manages the way algorithms are going to be fine-tuned or the performance of the algorithms.

It is a subscription based Service, provided as a Service in Azure now. In this post, you will know how likely Cognitive Services are of any help to you.

Tip #0 Ask (Convince) your boss

Microsoft Cognitive Services are tested against thousands (if not millions) of the users, data records, entities and the algorithms are concrete. You cannot meet the level, where Microsoft Cognitive Services are providing the Services, the reason is that Microsoft has partnered with quite a lot of academics’ professors, developers, teams and organizations and even most of the times, online surfers show up and share some data to Cloud — all of which are under a license and Microsoft asks for the permission but I am not here to cover up the license terms anyways.

Get the permissions, so that we may continue this post. 

Tip #1

Take only what you need

Cognitive Services are a library of Services. There are lot of Services, which are already added to the library and many are being added every month but this doesn’t mean, you should consider all of them or even half of them. They are all categorized under different sub sections, which contains collective Services provided by Microsoft CS,

• Vision
  
  
• This set of Services contains face APIs, such as recognition and tracking.
• It also provides Services, which can extract the features from faces such as age, emotion detection.
• It also provides computer vision, which can allow the users to perform OCR functions on the images.
  
  
• Speech
  
  
• It allows your users to trigger the functions, which are based on their vocal commands i.e. Natural language processing.
• Speaker Recognition — bleeding-edge technology.
• Speech to text and text to speech Services.
  
  
• Language
  
  
• It allows you to perform linguistic analysis of the text.
• You can use the previous Services to perform analysis on the photos as well, so that you read the text, using OCR and then analyze the text.
• LUIS (Language Understanding Intelligent Service) is new.
  
  
• Knowledge
  
  
• Recommender systems.
• Anything that requires complex academic, or research stuff.
  
  
• Search
  
  
• The old Bing APIs are now provided here…

Likewise, you can see that these are the categories and even these categories have different API sets and Services, which you might want to consume. It is up to you to select, which one you need.

In simple words, read the text from the images, convert them to speech and communicate. All you need to purchase is, “Computer Vision API”, and “Bing Speech API”. Your Application won’t need the rest of the Services. LUIS can be added finally to support the communication later.

There will be more Services and you can always add up more Services but if you are no longer using a Service or your Application is not related to a Service, there is no need to purchase a key for this Service.

Tip #2

Keep everything in Azure

Microsoft CS is provided from different areas (all Microsoft properties), such as LUIS and can be accessed through luis.ai and vice versa but you should keep the family tight and keep all the keys and resources in Azure. Thus, you can manage everything from a single subscription, instead of having to look at various accounts to configure and consume the Applications.

Microsoft CS supports REST-based API (and we will cover this in a later tip given below), so it is very easy to add the keys to the URL and start consuming the Services.

You can manage all the keys within Azure, just head over to the Cognitive Services blade and open the Application, which you want to get the keys for. Under the “Keys” section, look for the keys, which you can use to authenticate the requests.

Figure 1 List of the Cognitive Services associated with the account.

I have 4 services active, that I can access in the Azure through REST APIs. How simple that is! You can add more keys, add more services, update the keys… All from within Azure! By the end of this post, you will realize the importance of this tip.

Tip #3

Get the most out of REST API

Microsoft CS Azure endpoints are provided as REST API endpoints, then you can access it through any HTTP client — even a Web browser. The REST API, since working on HTTP protocol, allows you to make the best use of HTTP protocol and send/receive information. Currently, Microsoft CS supports two ways of uploading the information to the cloud,

1. URL based
2. Binary data base

These are the two ways that you can deliver the content to Azure for processing. Apart from this, the only required header for the request is the subscription key, added to the header of “Ocp-Apim-Subscription-Key“, which is processed first and the rest of the stuff is processed later, which is based on the subscription information.

Example

Now, let me show you a little example in WPF Application of consuming the Computer Vision API to detect what the image is all about. Azure will result in a complete sentence, which explains the image and the objects in the image as well as the task being done.

XAML code for WPF Application is given below.

<Grid>
    <Grid.ColumnDefinitions>
        <ColumnDefinition />
        <ColumnDefinition />
    </Grid.ColumnDefinitions>
    <Border BorderBrush="Black" BorderThickness="1" Width="211" Height="188">
        <Image Name="image" HorizontalAlignment="Left" Grid.Column="0" Height="188" MouseLeftButtonDown="Image_MouseLeftButtonDown" VerticalAlignment="Top" Width="211"/>
    </Border>
    <Button Name="btn" Grid.Column="0" Margin="0,0,24,10" Height="20" Width="70" Click="btn_Click" VerticalAlignment="Bottom" HorizontalAlignment="Right">Process</Button>
    <Button Name="slct" Grid.Column="0" Margin="24,0,0,10" Height="20" Width="70" Click="slct_Click" VerticalAlignment="Bottom" HorizontalAlignment="Left">Select</Button>
    <TextBlock Name="rslt" Margin="10" VerticalAlignment="Center" TextWrapping="Wrap" Grid.Column="1" Text="Result will be here..." />
</Grid>

Figure 2 WPF application running, with no image selected.

For the backend code, C# code was written, as shown below. 

private async void btn_Click(object sender, RoutedEventArgs e)
{
    using (var client = new HttpClient())
    {
        // Request headers
        client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", "<the-subscription-key>");

        // Request parameters
        var uri = $"https://westus.api.cognitive.microsoft.com/vision/v1.0/analyze?visualFeatures=Description";

        // Request body
        if(fileName == null) { MessageBox.Show("Select a file first."); }
        byte[] byteData = File.ReadAllBytes(fileName);

        using (var content = new ByteArrayContent(byteData))
        {
            content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
            var response = await client.PostAsync(uri, content);
        }

        rslt.Text = await response.Content.ReadAsStringAsync();
    }
}

private void slct_Click(object sender, RoutedEventArgs e)
{
    OpenFileDialog dialog = new OpenFileDialog();
    if (dialog.ShowDialog() == true)
    {
        // Something happened
        fileName = dialog.FileName;
        var source = new BitmapImage(new Uri(fileName));
        image.Source = source;
    }
}

Likewise, the output of this code, once worked was, as shown below.

Figure 3 Image selected and response captured from the Azure.

As seen, this is the result, which can be mapped to a JSON object for the storage or for further processing of the requests.

Tip #4

Timing is everything

Our interest in Microsoft CS is only possible, if it can guarantee that we get the results in a timely manner. For example, if we invest Microsoft CS in the security Applications, then the users should be provided with the results in a timely manner and the lagging may cause us to reconsider stuff around.

Thus, I wanted to show the time of the request as well to demonstrate, how it all works. For this, I modified the code and the changes given below were applied. 

Stopwatch watch = new Stopwatch();
watch.Start();
response = await client.PostAsync(uri, content);
watch.Stop();

rslt.Text = $"Request took {watch.ElapsedMilliseconds} ms to complete, for {byteData.Count()} sized byte array.\n\n";

rslt.Text += await response.Content.ReadAsStringAsync();

The effect of this was that I can determine how long, it does to process and return the result.

Figure 4 Application showing the time as well.

Look at the top paragraph, it says, “Request took 3519 milliseconds to complete, for 33282 sized byte array.” It means that to process a file of round about 30 KB, it took around 3.5 seconds. There are other factors, which caused the delay, such as my internet connection. Second, a larger image file will take more time and a smaller image will process quickly but with the errors.

There are few things, which we learn from this.

• The timing of Azure is not a big factor, the factors are given below.

• Our own internet connection
• The image itself

• Type of processing to be done is important
• Processing a sound track of 15 seconds with low quality vs a sound track of 1 min with high quality are never going to end up with same time.
• CDNs may or may not help in this case

Finally, the requirements differ from API to API, due to which  will not talk about the recommended image size but you can increase the performance of the Applications by uploading the files directly to Azure, because Azure is always going to download the file from the URL as well to process it. Thus, why not upload it directly?

Tip #5

Security

The keys for your Application are crucial. If they are lost or accessible to anyone, then you are responsible for what happens — in worse scenarios, they may use your own resources for their own use and you will be charged.

Remember Tip #2, if you followed my advice, you would now be able to easily change the key, if you feel someone has access to the keys.

Figure 5 Keys shown for the Microsoft Cognitive Service purchased from Azure.

You can also use other ways to hide the keys, if you don’t like updating the security keys every month. Some include storing the keys in secure areas, such as the Key Vault of Azure or any other place, where none can access but, what if someone does access it? 

In many ways, things can go wrong. Thus, it is my recommendation to update the keys every month. Note that you can use either Key 1 or Key 2 and you can update both the keys independently of each other.

Reminder

When you were reading this post, I went back and regenerated the keys, and it took only 4 seconds to regenerate them both. 

I hope you enjoyed the post. See you next time.