The inspiration for this interaction was to examine the direct translation from visual to audio. My goal was not to create representations of colors and shapes but to focus on directly converting color into sound. This was achieved by translating visual signals into electricity and connecting them to an audio mixer.

Using the ODD module from LZX Industries(Huge thanks to Tina Frank!), I was able to isolate different color channels, each of which has Minijack outputs. While these outputs are typically used to send visual signals, I connected them directly to the audio mixer. This setup enabled a direct translation from color to sound, revealing the immediate interaction between visual data and audio output.

An example of this process can be seen in Imagine This in a Big Room, where I primarily worked with green and alpha channels.




Later, I expanded this exploration using a USB microscope. I created sketches in TouchDesigner by separating color channels from the microscope’s output. By extracting values from the RGB channels, I was able to modulate three oscillators in real time. This experiment resulted in a digital color-to-audio transformation, achieved in real time.

Connecting pose detection via mediapipeline. 


Conclusion:
Yes, It works.
You cannot directly play happy birthday song.