How the Wacom Guitar Works

July 5, 2007 on 4:07 pm | In How the Wacom Guitar works

The Speech Guitar controls an application suite of standalone Patches made with Cycling 74’s graphical programming environment MaxMSP that includes Netochka Nezvanova’s Nato.0+55+3d (running on the OS 9 Macintosh platform). The applications utilise the Apple MacInTalk 3 speech synthesiser which is a text-to-speech (TTS) application. This needs to process text from a buffer embedded in the application or read data from linked external text files.

Throughout this project’s development there has been a couple of different ways of sourcing the text or lyrical content, the first coming from an abstraction that searches the internet for words or groups of words that describe or are related to emotions or feelings. The result is parsed from downloaded url pages and the html is extracted automatically. The data is sorted into lists of symbols contained in MaxMSP messages or it’s text object. This method produces mostly a random result of words but was used as the vocabulary needed for the Apple speech synth to work. This is Ok if there is a reliable ethernet or wireless connection and you are performing in a club or art space setting but not so practical for performances outside especially over distances from a wireless hub. Also the search function can introduce a timing latency from receiving gestural data from the Wacom tablet and it was decided to embed a text buffer in each standalone rather than to get the text in this way.

The other method was to manually search the internet for downloadable open source dictionary or vocabulary files and use a Perl script to parse the data into lists of symbols then copy and paste it into the MaxMSP code or into several external .txt files. With this is was possible to build extensive vocabularies of over 60 thousand words that could be instantly requested and read aloud from the TTS software. Each of the patchers vocabulary varied, some containing english word lists that described emotional states and feelings to attempting to get the Mac to sing in Japanese, Latin, or speak Gaelic phonetically which was perhaps a bit cheesy as the TTS voice that was chosen has an American accent with a deep voice but that said it is possible to get it to produce some interesting sounds.

Apple’s MacInTalk (TTS) voice “Bad News” is mostly used throughout in the software as it is the one that is most interesting to manipulate and is the most responsive to real-time request changes in pitch and rate. It is also a personal favorite as when this voice is processed via MSP algorithms it can suggest almost an emotional, chorale, quality. It is this that i have focused on during later stages of the projects development.

Different gestures drawn onto the Wacom graphics tablet trigger different parts of the process. The data from the tablet is mapped via the Max object Wacom by Richard Dudes to a series of if/then/else conditions and the object zmap for control of midi to hertz pitch, tts rate and parameter changes of MSP algorithms. Other gestures are recorded and can be looped using the LCD object, and mtr and so on. Each time the Intuos pen touches the surface of the tablet a word is instantly read from the text buffer, different gestures produce different ways that lists of symbols are formed. Single light taps with the pen on the surface of the Wacom produce singe words, flourishes can produce a looped layered sequences. Pitch and rate is dependent on which part of the tablet you draw upon.

There has been some development with OSX software for the Wacom Guitar controller by experimenting with other speech synthesis Max objects namely MaxMBOLA and Flite~. Although using the classic Macintosh platform has produced most of the body of work surrounding this project.

If time and personal direction allow i would like to further research and experiment in the area of interface design and musical expression through the performance of speech synthesis.