arrow-left-lightarrow-leftarrow-right-lightarrow-rightarrow-thin-left arrow-thin-right browser-search cup heart indicator laptop layers layout-4boxes layout-sideleft mail-heart mail map-pin mixer mouse nav paintbucket pencil-ruler phone picture play video
moriahgrace

Synesymph

  • Tag: AR
  • Medium: After Effects, Unity 3D, Vuforia Developer

“In 1842, when [Franz] Liszt took over the post of Kapellmeister in Weimar, he astonished the orchestra by saying, ‘Oh please, gentleman, a little bluer, if you please! This tone type requires it!’ Or, ‘That is a deep violet, please, depend on it. Not so rose!’ The orchestra eventually got used to the maestro seeing colors where they only saw notes.”  -Wednesday Is Indigo Blue by Richard E. Cytowic and David M. Eagleman

What is Synesymph?

Synesymph is a sound visualizing application I created for my senior thesis degree project. It uses Augumented Reality to simulate Synesthesia, a neurological condition in which one sense (in this case, hearing) involuntarily evokes another sense (in this case, sight). This is called Chromesthesia and is sometimes referred to as “colored hearing.”

How it works

Using 2D image targets, I took static visuals and brought them to life, syncing movement and color with audio. This was accomplished by working with the gaming software, Unity3D in tandem with the software development kit,  Vuforia.

Testing

Synesymph has proven to be an effective interactive piece. Below are a few reactions from the app's test run in the Polykhroma Gallery Series with Tufts University students.

Advocation & Launch

I created Gmail and Instagram campaigns to generate interest and increase attendance at Synesymph’s launch during my Thesis Symposium. 

Public Talks & Process

I presented my research, explaining what I learned about how pitch controls color while timbre controls movement and scintillation. To hear more about the science behind synesthesia as well as my design process, watch one of my talks below! 

Read All About It

Still curious? Take a look through the process book "Synesymph, Bring Sound to Sight".