Control your PC with your voice … and PowerShell

Joel 'Jaykul' Bennett — Thu, 25 Jun 2009 06:57:54 +0000

Have you ever wanted to be able to ask your computer questions and have it answer you out loud?
Have you ever wondered if your computer could be more like the ones running the Star Trek Enterprise, responding to voice queries and commands?

Have you played with ZWave or X10 home automation and thought that voice control of your devices would be an obvious next step?

Well, ok … I’m not going to show you how to turn on lights or work with home automation — but that’s the main thing that keeps me thinking about this voice-recognition stuff. What geek doesn’t want to walk into the living room and say “Computer: Lights On” and have it work?

Instead, as a first step to all of that, let me show you how to use PowerShell to do simple voice command recognition scripts … which can fire off any PowerShell script you care to write! The code which follows is really a module, although you can just dot-source it as a script, and it really requires PowerShell 2.0 for the events, although it would be trivial to refactor it to work on PowerShell 1.0 using Oisin’s PSEventing library.

$null = [Reflection.Assembly]::LoadWithPartialName("System.Speech")

## Create the two main objects we need for speech recognition and synthesis

if(!$Global:SpeechModuleListener){ ## For XP's sake, don't create them twice...

   $Global:SpeechModuleSpeaker = new-object System.Speech.Synthesis.SpeechSynthesizer

   $Global:SpeechModuleListener = new-object System.Speech.Recognition.SpeechRecognizer

}

$Script:SpeechModuleMacros = @{}

## Add a way to turn it off

$Script:SpeechModuleMacros.Add("Stop Listening", { $script:listen = $false; Suspend-Listening })

$Script:SpeechModuleComputerName = ${Env:ComputerName}

function Update-SpeechCommands {

#.Synopsis 

#  Recreate the speech recognition grammar

#.Description

#  This parses out the speech module macros, 

#  and recreates the speech recognition grammar and semantic results, 

#  and then updates the SpeechRecognizer with the new grammar, 

#  and makes sure that the ObjectEvent is registered.

   $choices = new-object System.Speech.Recognition.Choices

   foreach($choice in $Script:SpeechModuleMacros.GetEnumerator()) {

      New-Object System.Speech.Recognition.SemanticResultValue $choice.Key, 

                                                               $choice.Value.ToString() |

         ForEach-Object{ $choices.Add( $_.ToGrammarBuilder() ) }

   }

   if($VerbosePreference -ne "SilentlyContinue") { $Script:SpeechModuleMacros.Keys | 

      ForEach-Object { Write-Host "$Computer, $_" -Fore Cyan } }

   $builder = New-Object System.Speech.Recognition.GrammarBuilder "$Computer, "

   $builder.Append((New-Object System.Speech.Recognition.SemanticResultKey "Commands", 

                                                         $choices.ToGrammarBuilder()))

   $grammar = new-object System.Speech.Recognition.Grammar $builder

   $grammar.Name = "Power VoiceMacros"

   ## Take note of the events, but only once (make sure to remove the old one)

   Unregister-Event "SpeechModuleCommandRecognized" -ErrorAction SilentlyContinue

   $null = Register-ObjectEvent $grammar SpeechRecognized `

               -SourceIdentifier "SpeechModuleCommandRecognized" `

               -Action { iex $event.SourceEventArgs.Result.Semantics.Item("Commands").Value }

   $Global:SpeechModuleListener.UnloadAllGrammars()

   $Global:SpeechModuleListener.LoadGrammarAsync( $grammar )

}

function Add-SpeechCommands {

#.Synopsis

#  Add one or more commands to the speech-recognition macros, and update the recognition

#.Parameter CommandText

#  The string key for the command to remove

   [CmdletBinding()]

   Param([hashtable]$VoiceMacros,[string]$Computer=$Script:SpeechModuleComputerName)

   ## Add the new macros

   $Script:SpeechModuleMacros += $VoiceMacros 

   ## Update the default if they change it, so they only have to do that once.

   $Script:SpeechModuleComputerName = $Computer 

   Update-SpeechCommands

}

function Remove-SpeechCommands {

#.Synopsis

#  Remove one or more command from the speech-recognition macros, and update the recognition

#.Parameter CommandText

#  The string key for the command to remove

   Param([string[]]$CommandText)

   foreach($command in $CommandText) { $Script:SpeechModuleMacros.Remove($Command) }

   Update-SpeechCommands

}

function Clear-SpeechCommands {

#.Synopsis

#  Removes all commands from the speech-recognition macros, and update the recognition

#.Parameter CommandText

#  The string key for the command to remove

   $Script:SpeechModuleMacros = @{}

   ## Default value: A way to turn it off

   $Script:SpeechModuleMacros.Add("Stop Listening", { Suspend-Listening })

   Update-SpeechCommands

}

function Start-Listening {

#.Synopsis

#  Sets the SpeechRecognizer to Enabled

   $Global:SpeechModuleListener.Enabled = $true

   Say "Speech Macros are $($Global:SpeechModuleListener.State)"

   Write-Host "Speech Macros are $($Global:SpeechModuleListener.State)"

}

function Suspend-Listening {

#.Synopsis

#  Sets the SpeechRecognizer to Disabled

   $Global:SpeechModuleListener.Enabled = $false

   Say "Speech Macros are disabled"

   Write-Host "Speech Macros are disabled"

}

function Out-Speech {

#.Synopsis

#  Speaks the input object

#.Description

#  Uses the default SpeechSynthesizer settings to speak the string representation of the InputObject

#.Parameter InputObject

#  The object to speak 

#  NOTE: this should almost always be a pre-formatted string,

#        most objects don't render to very speakable text.

   Param( [Parameter(ValueFromPipeline=$true)][Alias("IO")]$InputObject )

   $null = $Global:SpeechModuleSpeaker.SpeakAsync(($InputObject|Out-String))

}

function Remove-SpeechXP {

#.Synopis

#  Dispose of the SpeechModuleListener and SpeechModuleSpeaker

   $Global:SpeechModuleListener.Dispose(); $Global:SpeechModuleListener = $null

   $Global:SpeechModuleSpeaker.Dispose(); $Global:SpeechModuleSpeaker = $null

}

set-alias asc Add-SpeechCommands

set-alias rsc Remove-SpeechCommands

set-alias csc Clear-SpeechCommands

set-alias say Out-Speech

set-alias listen Start-Listener

Export-ModuleMember -Function * -Alias * -Variable SpeechModuleListener, SpeechModuleSpeaker

There’s basically just one function you need to worry about here: New-VoiceCommands. You pass it a hashtable which maps strings to scriptblocks, and if you use the -Listen switch that’s all there is to it. You can also call Start-Listening manually, and of course, I’ve provided the Say function to make it easier to have the computer speak…

Once the computer is “listening” ... you just say it’s name, followed by one of your commands. I like that because it ensures that I don’t run the scripts by accident, but you can remove the "${Env:ComputerName}, " string from the beginning of the GrammarBuilder if you think it’s not necessary, or you can hard code it to something other than your computer’s name, like say “Hal, please, I beg you … “ or “Computer, please “ or whatever.

You can do a lot of things with this … anything, really … but to give you an example that you can easily understand, I’m going to do something very simple, and have my computer just answer a few basic questions by talking back to me, and then add a few commands to have it start an app or a web page.

Add-SpeechCommands @{

   "What time is it?" = { Say "It is $(Get-Date -f "h:mm tt")" }

   "What day is it?"  = { Say $(Get-Date -f "dddd, MMMM dd") }

   "What's running?"  = {

      $proc = ps | sort ws -desc

      Say $("$($proc.Count) processes, including $($proc[0].name), which is using " +

            "$([int]($proc[0].ws/1mb)) megabytes of memory")

   }

} -Computer "Laptop" -Verbose 

Add-SpeechCommands @{ "Run Notepad" = { &amp; "C:\Programs\DevTools\Notepad++\notepad++.exe" } }

Add-SpeechCommands @{ "Check Gee Mail" = { Start-Process "https://mail.google.com" } }

You see how easy that is? You can use “Say” to speak any text (although sometext will get better results than others), and you can invoke any other powershell commands, including HttpRest commands to fetch web data, or WASP commands for windows automation, or PowerBoots commands to display output in large text, or cmdlets to control X10 or ZWave devices … you know, anything

Huddled Masses » Speech recognition

Control your PC with your voice … and PowerShell