Demonstrating Synesthesia With LEGO

Translating colour to sound, all with LEGO and a bit of Python

For my first article in 2023, I wanted something unusual, yet familiar. The self-imposed requirement was, that it had to combine all the four major topics I usually write about: tech, coding, accessibility and LEGO. Not the easiest of tasks, but luckily for me, I had something in my back-pocket from way back in spring of 2020, when I “invented” something that, I felt, was pretty darn cool, but never got around to blog about it.

I guess, this is also the appropriate moment to admit a dirty little secret, that my earlier post about the LEGO smart lock, was, inspired by my earlier project, and half the code was, in fact, repurposed. Reusability for the win! But let’s not keep you on the edge of your toilet-seat — where you’re probably reading this — and let’s dive into a topic that I have been fascinated by for nearly two decades.

In 2020, I wanted to demonstrate synesthesia in a simple, playful and inspiring way. After a few hours of tinkering and coding, I did.

What is synesthesia?

I guess a quick intro into what it is and how I got introduced to it is going to make this story a tad more interesting. You will find that my first introduction to this perceptual phenomenon was in the NBC TV series called Heroes. One of the characters, Emma Coolidge — who also happens to be deaf, experiences what they called enhanced synesthesia, as she plays the cello (or maybe the double-base?). In hindsight, I think this was the moment the power of accessibility was planted into my subconscious. To this day, I think it’s one of the most powerful scenes from any TV show. I invite you to watch it, and another later scene demonstrating the phenomenon.

Synesthesia (American English) or synaesthesia (British English) is a perceptual phenomenon in which stimulation of one sensory or cognitive pathway leads to involuntary experiences in a second sensory or cognitive pathway. People who report a lifelong history of such experiences are known as synesthetes. — Wikipedia

There is virtually endless literature around the topic, so if you find it as fascinating as I do, I highly recommend doing some extra reading after this article.

Prerequisites for building the synesthesia demonstration

You’ll obviously need a few key components, but you’ll find that it’s a lot less complex compared to the smart lock project, so if you found that a tad intimidating, this should be more up your alley.

• An EV3 brick. This is essential. It cannot be another edition of LEGO’s smart brick, and you’ll see soon why. It has to be the one from set #31313 or the official education edition set #45544. You only need one smart brick for this prototype. You can also get it separately, but not for much longer.
• You’ll also ideally need a light/colour sensor. This will read the various colours from wherever you’d like it to, and allow us to translate that to a sound.
• Basic Python knowledge. I will perhaps soon write a companion article for this, introducing folks to the very basics of Python, but for now, you can learn these over on Codecademy or the many resources linked in this FreeCodeCamp article.
• A computer. It can be something as simple and cheap as a Raspberry Pi, and as premium and expensive as a Mac. A Windows machine will also do just fine. I used back in 2020 my 2019 Intel MacBook Air.
• A microSD or microSDHC card (2 GB or larger, though not larger than 32 GB — a limitation imposed by the EV3 brick).
• In terms of software, you’ll only need two things: VSCode and EV3dev. Additionally to VSCode, you’ll want to install the ev3dev-browser plugin. This will allow you to upload your project effortlessly to the brick.

See? Really not that much, and the fact that you don’t actually have to build anything out of LEGO bricks, apart from the smart brick and the light/colour sensor, makes this a very straight-forward project.

The synesthesia “engine”

While it’s much more common to see a certain colour based on a certain sound, I thought, in my demonstration, I would reverse it. The project is basic, and even in terms of coding, requires very little effort. The synesthesia “engine” is really just a glorified bunch of if-else statements, but hey, Apple taught us that marketing is important, so if I call it “engine”, you’d better believe it is! 😆

from ev3dev2.sensor import INPUT_1
from ev3dev2.sensor.lego import ColorSensor
from ev3dev2.sound import Sound

spkr = Sound()
xylophoneSensor = ColorSensor(INPUT_1)

spkr.speak('Starting synesthesia engine.')
timeRemaining = 100

while time > 0:
    # xylophone sensor runs on input port 1
    if xylophoneSensor.color_name == 'Blue':
        spkr.play_file('/home/robot/synesthesia/xylophone/a1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'Green':
        spkr.play_file('/home/robot/synesthesia/xylophone/b1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'Yellow':
        spkr.play_file('/home/robot/synesthesia/xylophone/c1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'Red':
        spkr.play_file('/home/robot/synesthesia/xylophone/d1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'Brown':
        spkr.play_file('/home/robot/synesthesia/xylophone/e1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'Black':
        spkr.play_file('/home/robot/synesthesia/xylophone/f1.wav')
        time = time - 1
    elif xylophoneSensor.color_name == 'White':
        spkr.play_file('/home/robot/synesthesia/xylophone/g1.wav')
        time = time - 1

If you check the code out, you’ll see we’re actually not doing that much. As the program starts, the smart brick tells the user the engine has started, at which point, it’s ready to consume input in the form of colour. For each colour the sensor can recognise, we output a different xylophone note. The instrument can be anything, really, as long as you have the required .wav files. One small note worth making, is that the sensor can actually recognise the lack of colour too, however, found that to increase unreliability, so I stuck to seven notes.

You can see a demonstration of the above code on my Instagram or TikTok. I’d love if you checked it out and followed me there, as I do post quite interesting LEGO and EV3 programming stuff there, stuff I don’t even put into articles. 😉

Speaking of seven notes, you can actually improve on this demo by quite a lot, though only if you employ an extra one or two light/colour sensors. That way you can use colour combinations to achieve full octaves and even polyphony. Should you want to modulate the sound, the first solution I can think of, is the ultrasonic sensor or the infrared sensor. You could also use a touch sensor to apply other effects or cycle between instruments. I mean, the sky and your imagination is really the only limit here.

Synesthesia is probably one of the coolest ways to introduce someone to diversity and accessibility. To open one’s eyes to just how differently we can all experience the same world.

Attila Vago — Software Engineer improving the world one line of code at a time. Cool nerd since forever, writer of codes and blogs. Web accessibility advocate, LEGO fan, vinyl record collector. Loves craft beer! Read my Hello story here! Subscribe and/or become a member for more stories about LEGO, tech, coding and accessibility! For my less regular readers, I also write about random bits and writing.