Programming Speech in WPF - Speech Recognition

Introduction

In my previous article Programming Speech in WPF - Speech Synthesis, I covered text-to-speech functionality in WPF. This article is about speech to text, also known as speech recognition.

Speech Recognition is a reverse process of Speech Synthesis that converts speech to text. There are two major applications for speech recognition. The first application is people who are for some reason unable to type but can speak to the system and system will type text for them. For example, in endoscopic applications a surgeon can evaluate the patient and speak to the system. While surgeon is doing the evaluation, his hands are buys but he can speak. The second application is speech command enabled applications where instead of using mouse, we can use voice to run and execute an application commands.

Windows Vista and Window 7 comes with built-in Speech Recognition controls that allow you to setup speech related options such as voice settings, microphone, and other voice recognition settings. Let's take a quick look at what Control Panel has to offer related to Speech Recognition.

Go to Control Panel and open Speech Recognition Options. You will see a dialog looks like Figure 1.



Figure 1

As you can see from Figure 5, there are options to start speech recognition, setup your microphone, take speech tutorial, train your computer, and open reference card. You may want to click on these options one by one to understand Speech Recognition better.

If you click on first link Start Speech Recognition, it will activate speech recognition on the system and system will start listening sounds around your computer.

Next option is Set up Microphone. This option allows you to tell system what microphone to use if you have more than one. Otherwise system will use default microphone.

Next option Take Speech Tutorial is a step by step tutorial that teaches you how to use various system controls.

Next option, Train your computer to better understand is very important. Before you want build and test your application, I recommend you use this option and follow step by steps of the wizard. This wizard will understand your voice and ensures the accuracy of commands you sends to the system. If you do not train your computer for your voice, computer may not understand your command properly.

The component that is responsible for controlling and managing speech recognition is called Windows Desktop Speech Technology Recognition Engine (SR Engine).

When you build a Speech Recognition application and you do not setup microphone and voice settings, system will launch wizards and it will ask you to setup these settings. On Windows Vista machine, when first time you will use its and some Speech Recognition controls, you will notice a Windows application like figure 2.



Figure 2

That tells me that SR Engine is ready. We just need to enable this by saying first command start listening.

If you right click on Speech Recognition control, you will see various options that allow you to turn speech recognition on, off and put it in sleep mode as you see in Figure 3.



Figure 3

Speech Recognition API

Speech Recognition functionality is defined in the System.Speech.Recognition namespace. Before you start using Speech Recognition related functionality, you must import these two namespaces in your application:

using System.Speech;

using System.Speech.Recognition;

SpeechRecognizer

The SpeechRecognizer is the main component of Speech Recognition API. The SpeechRecognition class is listens and catches the spoken text from the system and converts it to text or text commands.

protected SpeechRecognizer spRecognizer = new SpeechRecognizer();

Enabling SpeechRecognizer

The State property returns the current state of SpeechRecognizer that can either by in Stopped or Listening. The Enabled property controls if the SpeechRecognizer is enabled and ready to listen or not. Listing 9 enables SpeechRecognizer by setting Enabled property to true.

SpeechRecognizer spRecognizer = new SpeechRecognizer();

spRecognizer.Enabled = true;

Listing 9

Reading Text

The SpeechRecignized event of SpeechRecognizer is raised when the recognition engine detects speech, and has found one or more phrases with sufficient confidence levels. This event is used to get the speech that is detected by the speech engine.

The code snippet in Listing 10 sets the SpeechRecognized event handler and gets the text recognized by the speech engine and copies it in a string.

SpeechRecognizer  spRecognizer = new SpeechRecognizer();

spRecognizer.Enabled = true;

spRecognizer.SpeechRecognized += new EventHandler<SpeechRecognizedEventArgs>(spRecognizer_SpeechRecognized); 

 

void spRecognizer_SpeechRecognized(object sender, SpeechRecognizedEventArgs e)

{

    string str = e.Result.Text;

}

Listing 10

Grammar and GrammarBuilder

One of the key usages of speech-enabled applications to build software product that listens to your commands and execute functionality based on the given commands. For example, instead of using a menu items to open and close files, we can build a system that will open and close a file when speech command Open File and Close File are sent to the speech system. The Grammar object of SpeechRecognizer handles these commands and the Grammar class is used to create a Grammar component.

The Grammar object in WPF represents a grammar document. The Grammar object fully supports the W3C Speech Recognition Grammar Specification (SRGS) and Context Free Grammar (CFG) specifications. You create a Grammar object by passing a GrammarBuilder object as a parameter in its constructor. Listing 11 creates a Grammar object by passing a Grammer Builder object as the default parameter of its constructor.
 

GrammarBuilder gBuilder = new GrammarBuilder();

// Construct GrammarBuilder here

 

// Create a Grammar from a GrammarBuilder

Grammar speechGrammar = new Grammar(gBuilder);

Listing 11

A GrammarBuilder object is used to provide a simple mechanism to build speech grammar. Add and Append methods of GrammarBuilder are used to add and append speech text, phrases and other GrammarBuilder objects to a grammar.

The methods of GrammarBuilder take parameters of either string or Choices object. The Choices object represents a list of alternative items to make up an element in a speech grammar.

The code snippet in Listing 12 creates a Grammar Builder using some Choices objects and then builds a Grammar object that can be load into a SpeechRecognizer.
 

private Grammar CreateGrammarDocument()

{

    GrammarBuilder gBuilder = new GrammarBuilder();

    // Construct GrammarBuilder here

    gBuilder.Append(new Choices("Phone", "Email", "Text"));

    gBuilder.Append("my");

    gBuilder.Append(new Choices("Mom", "Dad", "Brother", "Sister"));

 

    // Create a Grammar from a GrammarBuilder

    Grammar speechGrammar = new Grammar(gBuilder);

    return speechGrammar;

}

Listing 12

Here is a list of few sentences that can be constructed using Listing 12.

• Phone my Mom
• Text my Brother
• Email my Mom
• Phone my Brother
• Email my Dad

Loading and Unloading Grammar

The LoadGrammar method of SpeechRecognizer synchronously loads a specific grammar into a SpeechRecognizer. The code snippet in Listing 13 calls LoadGrammar method and loads a grammar.

SpeechRecognizer  spRecognizer = new SpeechRecognizer();

spRecognizer.LoadGrammar(CreateGrammarDocument());

Listing 13

The LoadGrammarSync method of SpeechRecognizer asynchronously loads a specific grammar into a SpeechRecognizer. The code snippet in Listing 14 calls LoadGrammarAsync method and loads a grammar.

 

SpeechRecognizer  spRecognizer = new SpeechRecognizer();

spRecognizer.LoadGrammarAsync(CreateGrammarDocument());

Listing 14

The UnloadGrammar method unloads a given Grammar and UnloadAllGrammars method unloads all grammars in a SpeechRecognizer object. The code snippet in Listing 15 shows how to upload grammars using UnloadGrammar and UnloadAllGrammars methods.

spRecognizer.UnloadGrammar(g);
spRecognizer.UnloadAllGrammars();

Listing 15

SRGS

Speech Recognition Grammar Specification (SRGS) is a W3C recommendation to build grammar that is used in speech enabled applications. More details about SRGS can be found at http://www.w3.org/TR/speech-grammar/.

The System.Speech.Recognition.SrgsGrammar namespace defines all functionality related to SRGS. The SrgsDocument class represents a SRGS document. The namespace also have classes for grammar objects such as SrgsElement, SgrsItem, SrgsOneOf, SrgsRule, SrgsText, SrgsToken and so on. In WPF, each object has its own class. Discussion of these classes in details is out of scope of this chapter.

The following code snippet creates a Rule and sets its scope.
 

SrgsRule rootRule = new SrgsRule("Months and Days");

rootRule.Scope = SrgsRuleScope.Public;

The following code snippet adds an element to a Rule.

rootRule.Elements.Add(new SrgsItem("Months and Days Grammar "));

And the following code snippet adds a rule to a document.
 

SrgsText textItem = new SrgsText("Start of the Document.");

SrgsRule textRule = new SrgsRule("TextItem");

textRule.Elements.Add(textItem);

document.Rules.Add(textRule);

Listing 16 creates a complete SRGS document dynamically and saves this document in an XML file. As you can see from Listing 16, the code adds rules for months and days of week and some extra items as rules.
 

private SrgsDocument BuildDynamicSRGSDocument()

{

    // Create SrgsDocument

    SrgsDocument document = new SrgsDocument();

 

    // Create Root Rule

    SrgsRule rootRule = new SrgsRule("MonthsandDays");

    rootRule.Scope = SrgsRuleScope.Public;

 

    rootRule.Elements.Add(new SrgsItem("Months and Days Grammar "));

 

    // Create months

    SrgsOneOf oneOfMonths = new SrgsOneOf(

        new SrgsItem("January"),

        new SrgsItem("February"),

        new SrgsItem("March"),

        new SrgsItem("April"),

        new SrgsItem("May"),

        new SrgsItem("June"),

        new SrgsItem("July"),

        new SrgsItem("August"),

        new SrgsItem("September"),

        new SrgsItem("October"),

        new SrgsItem("November"),

        new SrgsItem("December")

        );

    SrgsRule ruleMonths = new SrgsRule("Months", oneOfMonths);

    SrgsItem of = new SrgsItem("of");

    SrgsItem year = new SrgsItem("year");

    SrgsItem ruleMonthsItem = new SrgsItem(new SrgsRuleRef(ruleMonths), of, year);

 

    // Create Days

    SrgsOneOf oneOfDays = new SrgsOneOf(

        new SrgsItem("Monday"),

        new SrgsItem("Tuesday"),

        new SrgsItem("Wednesday"),

        new SrgsItem("Thursday"),

        new SrgsItem("Friday"),

        new SrgsItem("Saturday"),

        new SrgsItem("Sunday")

        );

    SrgsRule ruleDays = new SrgsRule("Days", oneOfDays);

    SrgsItem week = new SrgsItem("week");

    SrgsItem ruleDaysItem = new SrgsItem(new SrgsRuleRef(ruleDays), of, week);

 

    // Add items to root Rule

    rootRule.Elements.Add(ruleMonthsItem);

    rootRule.Elements.Add(ruleDaysItem);

 

 

    // Add all Rules to Document

    document.Rules.Add(rootRule, ruleMonths, ruleDays);

  // Add some extra sperate Rules

    SrgsText textItem = new SrgsText("Start of the Document.");

    SrgsRule textRule = new SrgsRule("TextItem");

    textRule.Elements.Add(textItem);

    document.Rules.Add(textRule);

 

    SrgsItem stringItem = new SrgsItem("Item as String.");

    SrgsRule itemRule = new SrgsRule("ItemRule");

    itemRule.Elements.Add(stringItem);

    document.Rules.Add(itemRule);

 

    SrgsItem elementItem = new SrgsItem();

    SrgsRule elementRule = new SrgsRule("ElementRule");

    elementRule.Elements.Add(elementItem);

    document.Rules.Add(elementRule);

 

    // Set Document Root

    document.Root = rootRule;

 

    // Save Created SRGS Document to XML file

    XmlWriter writer = XmlWriter.Create("DynamicSRGSDocument.Xml");

    document.WriteSrgs(writer);

    writer.Close();

 

    return document;

}

Listing 16

The document generated by code Listing 16 looks like Figure 4.



Figure 4

We can load a SRGS document as a parameter in the Grammar constructor to create a grammar from a SrgsDocument. The code snippet in Listing 17 loads a SRGS Grammar document by calling SpeechRecognizer's LoadGrammar.
 

SpeechRecognizer spRecognizer = new SpeechRecognizer();

spRecognizer.LoadGrammar(new Grammar(BuildDynamicSRGSDocument()));

Listing 17

Summary

Speech API (SAPI) 5.3 is a managed API comes with Windows Vista. This chapter demonstrated how we can use SAPI in a WPF application to build speech-enabled applications. First part of this article covered the text-to-speech (TTS) or Speech Synthesis, Programming Speech in WPF - Speech Synthesis where we built an application that convert text to speech. The second part of the article discussed speech recognition where we built an application that captures the speech from a voice device and convert to text. We also saw how to build speech grammars and use these grammars in speech-enabled applications.