Customer.ioAutomate Messaging Everywhere — Startups Get 12 Months Free
Promoted
gesture.live
Play live electronic music with your hands using your webcam
114 followers
Play live electronic music with your hands using your webcam
114 followers
Play live electronic music with your hands over a webcam. Chords, bass, drums and effects, performed through gestures.
gesture.live
Shoutout to indecisiveeric.com/gesture-synth, which went viral a few weeks ago on Instagram. It allows users to use both hands to perform chords.
gesture.live is an exploration on how to take this gesture synth further. I am using the same chord language, but am compressing it to one hand, so the other hand is free to play melodies, add drum/bass patterns and control effects.
The question that decides this is whether note-ons are quantised. Webcam hand tracking is 30fps on a good day, so there's 30ms of jitter before the audio thread even hears you, and you start feeling timing errors well under that. Snapping to a 16th grid makes it playable but you lose pushing and laying back, which is most of what makes a part sound human. Freeing the second hand by compressing chords into one is the right call either way.
gesture.live
@asadmalik901 I am quantizing the chords to the grid since they also influence the bass and the notes available to play the melody. The melody is completely in realtime! I haven't found latency a big issue so far, but its good to to think about!
@robin_van_soelen Splitting it that way is smarter than quantising everything, the chords carry the harmony and the melody carries the feel. The thing I'd watch is that latency doesn't bother you because you built the mapping, your hands already know where the notes are before you move them. Someone learning it reads the same 30ms as the instrument being unresponsive rather than themselves being early, and that's the moment they close the tab.
Dial
the demo video looks great in a well lit room at a desk. has anyone actually tried this on a small stage or at a house party yet? I'm curious how the hand tracking holds up under stage lighting or just dimmer, more colored light than a webcam usually gets pointed at, since that seems like the real test of whether this leaves the bedroom-producer use case.
gesture.live
@galdayan Thanks for the comment! I haven't tested it with a live audience yet. The initial use case was for the bedroom, but would love to optimize for live performance in the future!
Dial
@robin_van_soelen makes sense to ship for the use case in front of you first. stage lighting is a solvable problem later, and probably an easier one than the realtime chord tracking you already nailed - worth keeping on the roadmap once the bedroom version has legs.
I am no musician, so probably I will not ever use this, other than to play with my kids, but this seems amazing!
gesture.live
@igor_torrealba Thank you! I hope your kids will enjoy it :)