UWP - Retrieving Image Features Using Cognitive Service Computer Vision API

Before reading this article, please go through the following article.

1. Introduction To Universal Windows Platform (UWP) App Development Using Windows 10 And Visual Studio 2015
2. Getting Started With Microsoft Azure Cognitive Services - Computer Vision API

Microsoft Cognitive Services (formerly Project Oxford) are a set of APIs, SDKs and services available to developers to make their applications more intelligent, engaging and discoverable. Microsoft Cognitive Services expand on Microsoft’s evolving portfolio of machine learning APIs and enables developers to easily add intelligent features. Microsoft Cognitive Services let you build apps with powerful algorithms using just a few lines of code. They work across devices and platforms, such as iOS, Android, and Windows, keep improving, and are easy to set up.

Computer Vision API, as described by Microsoft, extracts rich information from the images to categorize and process the visual data and protect your users from unwanted content. It supports, analyzing an image, generating thumbnail, tagging, describing , reading text in the images and recognizing.

Reading this article, you'll learn how to retrieve image features like Image Properties, Tags, Describing Image from the selected image using Cognitive Service Computer Vision API in Universal Windows Apps development with Azure, XAML and Visual C#.

The following important tools are required for developing UWP.

1. Windows 10 (Recommended)
2. Visual Studio 2015 Community Edition (It is a free software available online)
3. Cognitive Service Computer Vision API Key

Now, we can discuss step by step app development.

Step 1

Open Visual Studio 2015 -> Start -> New Project-> Select Universal (under Visual C# -> Windows) -> Blank App -> Give suitable name for your App (UWPCogVisDesImg) -> OK.



After choosing the target and minimum platform version that your app will support, the Project creates App.xaml and MainPage.xaml.



Step 2

Open (double click) the file MainPage.xaml in the Solution Explorer and add the Microsoft.ProjectOxford.Vision reference in the project. Right click your project (UWPCogVisDesImg) and select "Manage NuGet Packages".



Choose "Browse" and Search "Microsoft.ProjectOxford.Vision". Select the package and install it.



Reference is added to your project.



Step 3

Add a TextBlock control, change the name and text property for Title.



Add a Button Control, set the Name for selecting an Image.



Add an Image control, change the name and text property for displaying the selected image.



Add a Button Control, set the name for getting the Image features.



Add a TextBlock control, change the name, and clear the text property for displaying Description Result.



Add a TextBlock control, change the name, and clear the text property for displaying Tags Result.



Add a TextBlock control, change the name and clear the text property for displaying Image Properties Result.



Add a Click event method for both the button controls.



Note

Automatically, the following code will be generated in XAML code View, while we are done in the design view.

<Page x:Class="UWPCogVisDesImg.MainPage" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:local="using:UWPCogVisDesImg" xmlns:d="http://schemas.microsoft.com/expression/blend/2008" xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006" mc:Ignorable="d">
    <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        <TextBlock x:Name="tblTitle" HorizontalAlignment="Left" Margin="348,70,0,0" TextWrapping="Wrap" Text="UWP Cognitive Service Computer Vision Features" VerticalAlignment="Top" FontWeight="Bold" FontSize="20" />
        <Button x:Name="btnSel" Content="Select Your Image" HorizontalAlignment="Left" Margin="322,152,0,0" VerticalAlignment="Top" Click="btnSel_Click" />
        <Image x:Name="imgDescribe" HorizontalAlignment="Left" Height="356" Margin="84,218,0,0" VerticalAlignment="Top" Width="377" />
        <Button x:Name="btnCVFeature" Content="Get Features" HorizontalAlignment="Left" Margin="592,320,0,0" VerticalAlignment="Top" FontWeight="Bold" Click="btnCVFeature_Click" />
        <TextBlock x:Name="tblDescribe" HorizontalAlignment="Left" Margin="719,199,0,0" TextWrapping="Wrap" Text="" VerticalAlignment="Top" Height="76" Width="366" FontSize="18" />
        <TextBlock x:Name="tblTags" HorizontalAlignment="Left" Margin="1132,165,0,0" TextWrapping="Wrap" Text="" VerticalAlignment="Top" Height="456" Width="138" FontSize="18" />
        <TextBlock x:Name="tblImgtype" HorizontalAlignment="Left" Margin="847,389,0,0" TextWrapping="Wrap" Text=" " VerticalAlignment="Top" Height="147" Width="280" /> </Grid>
</Page>

Step 4

Add the following namespaces in Mainpage.xaml.cs for Tagging Images.

using Windows.Storage;
using Windows.UI.Xaml.Media.Imaging;
using Windows.Storage.Pickers;
using System.Threading.Tasks;
using Windows.UI.Popups;
using Windows.ApplicationModel;
using Microsoft.ProjectOxford.Vision.Contract;
using Microsoft.ProjectOxford.Vision;

Step 5

Add the Cognitive Service Computer Vision API Client keys use Azure service and Generate it (For More Information, Please refer the article Getting Started With Microsoft Azure Cognitive Services - Computer Vision API.

var visionClient = new VisionServiceClient("<Your Key>");
Add the following code with azure generated key
for getting Image Details,
StorageFile file;
public string Widt {
    get;
    set;
}
public string Heig {
    get;
    set;
}
public string Format {
    get;
    set;
}
public string ClipArtType {
    get;
    set;
}
public string LineDrawingType {
    get;
    set;
}
private async void btnSel_Click(object sender, RoutedEventArgs e) {
    file = await FilePicker.cur.GetImage();
    if (file != null) {
        var im = new BitmapImage();
        using(var straem = await file.OpenAsync(FileAccessMode.Read)) {
            im.SetSource(straem);
            imgDescribe.Source = im;
        }
    }
}
static async Task < AnalysisResult > getDescribe(Stream s) {
    var visionClient = new VisionServiceClient("<Your Key>");
    var features = new VisualFeature[] {
        VisualFeature.Categories, VisualFeature.Color, VisualFeature.Description, VisualFeature.ImageType, VisualFeature.Tags
    };
    AnalysisResult Result = await visionClient.AnalyzeImageAsync(s, features);
    return Result;
}
private async void btnCVFeature_Click(object sender, RoutedEventArgs e) {
    using(var stream = await file.OpenStreamForReadAsync()) {
        var resul = await getDescribe(stream);
        foreach(var caption in resul.Description.Captions) {
            tblDescribe.Text = "Description : \n" + "Caption : " + caption.Text + "\n Confidence : " + caption.Confidence + "\n";
        }
        string tags = "Tags : \n";
        foreach(var tag in resul.Description.Tags) {
            tags += tag + "\n";
        }
        tblTags.Text = tags;
        Widt = "Image Width : " + resul.Metadata.Width;
        Heig = "\n Image Height : " + resul.Metadata.Height;
        Format = "\n Image Format : " + resul.Metadata.Format;
        ClipArtType = "\n ClipArtType : " + resul.ImageType.ClipArtType;
        LineDrawingType = "\nLineDrawingType: " + resul.ImageType.LineDrawingType;
        string clipArtType;
        switch (resul.ImageType.ClipArtType) {
            case 0:
                clipArtType = "\n ClipArtType : 0 Non-clipart";
                break;
            case 1:
                clipArtType = "\n ClipArtType : 1 ambiguous";
                break;
            case 2:
                clipArtType = "\n ClipArtType : 2 normal-clipart";
                break;
            case 3:
                clipArtType = "\n ClipArtType : 3 good-clipart";
                break;
            default:
                clipArtType = "\n ClipArtType : Unknown";
                break;
        }
        tblImgtype.Text = "Image Properties : \n" + Widt + Heig + Format + clipArtType + LineDrawingType;
    }
}
}
class FilePicker {
    private FileOpenPicker pick;
    private static FilePicker inst;
    public static FilePicker cur {
        get {
            return inst;
        }
    }
    static FilePicker() {
        inst = new FilePicker();
    }
    private FilePicker() {
        pick = new FileOpenPicker();
        pick.FileTypeFilter.Add(".png");
        pick.FileTypeFilter.Add(".jpg");
        pick.FileTypeFilter.Add(".gif");
        pick.FileTypeFilter.Add(".bmp");
        pick.FileTypeFilter.Add(".jpeg");
    }
    public async Task < StorageFile > GetImage() {
        repeat: StorageFile s = await pick.PickSingleFileAsync();
        var prp = await s.GetBasicPropertiesAsync();
        if (prp.Size > 4000000) {
            MessageDialog dialog = new MessageDialog("File size should be greater than 4 MB");
            await dialog.ShowAsync();
            s = null;
            goto repeat;
        }
        return s;
    }
    public async Task < StorageFile > GetFile(string filename, string foldername) {
        StorageFolder appFolder = Package.Current.InstalledLocation;
        StorageFolder folder = await appFolder.GetFolderAsync(foldername);
        var file = await folder.GetFileAsync(filename);
        return file;
    }
}

Step 6

Deploy your app in Local Machine. The output of the UWPCogVisDesImg App is shown below.



Another image with description.



Summary

Now, you have successfully tested the selected image using Cognitive Service Vision API - Describe Image, Azure, XAML AND C# with UWP environment.