Browse: Home / Control your PC with your voice … and PowerShell

Control your PC with your voice … and PowerShell

By Joel 'Jaykul' Bennett on 25-Jun-2009

  • Have you ever wanted to be able to ask your computer questions and have it answer you out loud?
  • Have you ever wondered if your computer could be more like the ones running the Star Trek Enterprise, responding to voice queries and commands?
  • Have you played with ZWave or X10 home automation and thought that voice control of your devices would be an obvious next step?

Well, ok … I’m not going to show you how to turn on lights or work with home automation — but that’s the main thing that keeps me thinking about this voice-recognition stuff. What geek doesn’t want to walk into the living room and say “Computer: Lights On” and have it work?

Instead, as a first step to all of that, let me show you how to use PowerShell to do simple voice command recognition scripts … which can fire off any PowerShell script you care to write! The code which follows is really a module, although you can just dot-source it as a script, and it really requires PowerShell 2.0 for the events, although it would be trivial to refactor it to work on PowerShell 1.0 using Oisin’s PSEventing library.


$null = [Reflection.Assembly]::LoadWithPartialName("System.Speech")

## Create the two main objects we need for speech recognition and synthesis
if(!$Global:SpeechModuleListener){ ## For XP's sake, don't create them twice...
   $Global:SpeechModuleSpeaker = new-object System.Speech.Synthesis.SpeechSynthesizer
   $Global:SpeechModuleListener = new-object System.Speech.Recognition.SpeechRecognizer
}

$Script:SpeechModuleMacros = @{}
## Add a way to turn it off
$Script:SpeechModuleMacros.Add("Stop Listening", { $script:listen = $false; Suspend-Listening })
$Script:SpeechModuleComputerName = ${Env:ComputerName}

function Update-SpeechCommands {
#.Synopsis
#  Recreate the speech recognition grammar
#.Description
#  This parses out the speech module macros,
#  and recreates the speech recognition grammar and semantic results,
#  and then updates the SpeechRecognizer with the new grammar,
#  and makes sure that the ObjectEvent is registered.
   $choices = new-object System.Speech.Recognition.Choices
   foreach($choice in $Script:SpeechModuleMacros.GetEnumerator()) {
      New-Object System.Speech.Recognition.SemanticResultValue $choice.Key,
                                                               $choice.Value.ToString() |
         ForEach-Object{ $choices.Add( $_.ToGrammarBuilder() ) }
   }

   if($VerbosePreference -ne "SilentlyContinue") { $Script:SpeechModuleMacros.Keys |
      ForEach-Object { Write-Host "$Computer, $_" -Fore Cyan } }

   $builder = New-Object System.Speech.Recognition.GrammarBuilder "$Computer, "
   $builder.Append((New-Object System.Speech.Recognition.SemanticResultKey "Commands",
                                                         $choices.ToGrammarBuilder()))
   $grammar = new-object System.Speech.Recognition.Grammar $builder
   $grammar.Name = "Power VoiceMacros"

   ## Take note of the events, but only once (make sure to remove the old one)
   Unregister-Event "SpeechModuleCommandRecognized" -ErrorAction SilentlyContinue
   $null = Register-ObjectEvent $grammar SpeechRecognized `
               -SourceIdentifier "SpeechModuleCommandRecognized" `
               -Action { iex $event.SourceEventArgs.Result.Semantics.Item("Commands").Value }
   
   $Global:SpeechModuleListener.UnloadAllGrammars()
   $Global:SpeechModuleListener.LoadGrammarAsync( $grammar )
}

function Add-SpeechCommands {
#.Synopsis
#  Add one or more commands to the speech-recognition macros, and update the recognition
#.Parameter CommandText
#  The string key for the command to remove
   [CmdletBinding()]
   Param([hashtable]$VoiceMacros,[string]$Computer=$Script:SpeechModuleComputerName)
   
   ## Add the new macros
   $Script:SpeechModuleMacros += $VoiceMacros
   ## Update the default if they change it, so they only have to do that once.
   $Script:SpeechModuleComputerName = $Computer
   Update-SpeechCommands
}

function Remove-SpeechCommands {
#.Synopsis
#  Remove one or more command from the speech-recognition macros, and update the recognition
#.Parameter CommandText
#  The string key for the command to remove
   Param([string[]]$CommandText)
   foreach($command in $CommandText) { $Script:SpeechModuleMacros.Remove($Command) }
   Update-SpeechCommands
}

function Clear-SpeechCommands {
#.Synopsis
#  Removes all commands from the speech-recognition macros, and update the recognition
#.Parameter CommandText
#  The string key for the command to remove
   $Script:SpeechModuleMacros = @{}
   ## Default value: A way to turn it off
   $Script:SpeechModuleMacros.Add("Stop Listening", { Suspend-Listening })
   Update-SpeechCommands
}


function Start-Listening {
#.Synopsis
#  Sets the SpeechRecognizer to Enabled
   $Global:SpeechModuleListener.Enabled = $true
   Say "Speech Macros are $($Global:SpeechModuleListener.State)"
   Write-Host "Speech Macros are $($Global:SpeechModuleListener.State)"
}
function Suspend-Listening {
#.Synopsis
#  Sets the SpeechRecognizer to Disabled
   $Global:SpeechModuleListener.Enabled = $false
   Say "Speech Macros are disabled"
   Write-Host "Speech Macros are disabled"
}

function Out-Speech {
#.Synopsis
#  Speaks the input object
#.Description
#  Uses the default SpeechSynthesizer settings to speak the string representation of the InputObject
#.Parameter InputObject
#  The object to speak
#  NOTE: this should almost always be a pre-formatted string,
#        most objects don't render to very speakable text.
   Param( [Parameter(ValueFromPipeline=$true)][Alias("IO")]$InputObject )
   $null = $Global:SpeechModuleSpeaker.SpeakAsync(($InputObject|Out-String))
}

function Remove-SpeechXP {
#.Synopis
#  Dispose of the SpeechModuleListener and SpeechModuleSpeaker
   $Global:SpeechModuleListener.Dispose(); $Global:SpeechModuleListener = $null
   $Global:SpeechModuleSpeaker.Dispose(); $Global:SpeechModuleSpeaker = $null
}

set-alias asc Add-SpeechCommands
set-alias rsc Remove-SpeechCommands
set-alias csc Clear-SpeechCommands
set-alias say Out-Speech
set-alias listen Start-Listener
Export-ModuleMember -Function * -Alias * -Variable SpeechModuleListener, SpeechModuleSpeaker

There’s basically just one function you need to worry about here: New-VoiceCommands. You pass it a hashtable which maps strings to scriptblocks, and if you use the -Listen switch that’s all there is to it. You can also call Start-Listening manually, and of course, I’ve provided the Say function to make it easier to have the computer speak…

Once the computer is “listening” ... you just say it’s name, followed by one of your commands. I like that because it ensures that I don’t run the scripts by accident, but you can remove the "${Env:ComputerName}, " string from the beginning of the GrammarBuilder if you think it’s not necessary, or you can hard code it to something other than your computer’s name, like say “Hal, please, I beg you … “ or “Computer, please “ or whatever. :-)

You can do a lot of things with this … anything, really … but to give you an example that you can easily understand, I’m going to do something very simple, and have my computer just answer a few basic questions by talking back to me, and then add a few commands to have it start an app or a web page.


Add-SpeechCommands @{
   "What time is it?" = { Say "It is $(Get-Date -f "h:mm tt")" }
   "What day is it?"  = { Say $(Get-Date -f "dddd, MMMM dd") }
   "What's running?"  = {
      $proc = ps | sort ws -desc
      Say $("$($proc.Count) processes, including $($proc[0].name), which is using " +
            "$([int]($proc[0].ws/1mb)) megabytes of memory")
   }
} -Computer "Laptop" -Verbose

Add-SpeechCommands @{ "Run Notepad" = { & "C:\Programs\DevTools\Notepad++\notepad++.exe" } }
Add-SpeechCommands @{ "Check Gee Mail" = { Start-Process "https://mail.google.com" } }
 

You see how easy that is? You can use “Say” to speak any text (although sometext will get better results than others), and you can invoke any other powershell commands, including HttpRest commands to fetch web data, or WASP commands for windows automation, or PowerBoots commands to display output in large text, or cmdlets to control X10 or ZWave devices … you know, anything ;-)

Reblog this post [with Zemanta]

Similar Posts:

Posted in | Tagged , , , , | 5 Responses

5 responses to “Control your PC with your voice … and PowerShell”

  1. Andy
    Andy 25-Jun-2009 at 7:39 pm |

    Export-ModuleMember -Function * -Alias * -Variable SpeachModuleListener, SpeechModuleSpeaker

    An ‘a’ in speach above ?

    Thanx as ever Jaykul
    Andy

    1. Joel 'Jaykul' Bennett
      Joel 'Jaykul' Bennett 25-Jun-2009 at 8:54 pm |

      Thanks, yes. I’ve corrected it :)

  2. Hal Rottenberg
    Hal Rottenberg 29-Jun-2009 at 7:45 am |

    Cue jokes about me here…

  3. Andreas
    Andreas 06-Jul-2009 at 5:40 pm |

    It would be very cool if you can add something like “Computer query %a” where %a gets send to wolfram|alpha and the result is being said.

    1. Joel 'Jaykul' Bennett
      Joel 'Jaykul' Bennett 07-Jul-2009 at 12:25 am |

      That blog post is coming very soon ;) but since you asked so nicely … I pasted a version of the script which supports that here http://poshcode.org/1195 — the key is that you can put a wildcard in, and then use $match in your script … like:

      Add-SpeechCommands  @{
         "Define * please" = {
            $term = $_ -replace "$Computer, Define (.*) please",''
            Say (Get-Definition $term)
         }
      }

      For now, I’ll leave it up to you to write the implementation of Get-Definition ;) ... but bear in mind that the * can match more than one word, so it’s up to you to either write a sensible script … or speak carefully ;)a