This week Yilin Zou and I set out to make some sound machines. We had too many ideas to nail it down to just one. So we decided to make a few sketches testing out our various ideas. We had two goals: to make something using poseNet and the pixel array in p5.js to see kinds of sounds we can make using a webcam. Here's what we came up with:
Pixel Array Sound Machine
This sketch uses the pixel array to create sound. The original goal was to make a rain machine that responded to motion on the screen, but after doing some research we were having a hard time figuring out how to make the exact sound we were looking for. Instead we discovered p5 filters which allowed us to amplify and modulate certain frequencies using an FFT object. I'm still not a hundred percent sure how this code works but by using p5.noise() and p5.BandPass(), we were able to generate the initial "white-noise" sound we were looking for. From there we mapped the pixel array RGB values to the filter frequency and filter width value range. It took a lot of tweaking but I was eventually able to get the sound to hopefully not blow out any speakers or eardrums and I like what we ended up with. It doesn't sound like rain, but you can hear a drop here and there. We matched the sounds to the video output with some ellipses and a simple grayscale color scheme for a nice visual effect.
We also explored separately on the posenet to learn the basics. Our idea is to make some buttons on the screen so players can play different keys using their wrists.
Then we tried the coding with three keys.
What Didn't Work/What We Learned
Through our experimentation, posenet is not as stable as we expected, so we also tried to display music with visual designs. However we ran into a few issues getting the animations to work the way we wanted them to. The pixel array was offset in our first sketch, which we still can't seem to figure out. In this sketch, players can type on different keys(A,S,D,F,G,spacebar and J,K,L,; ) to play music along with the background music, but we were having trouble properly animating the music with the movement of an ellipse. Thanks to Lisa’s help, we were able to see the circles drawn by the background music. However, we still have trouble receiving the speaker’s sound and the background sound at the same time. You can see the sketch below.