Tap to take a photo! The height of the tap determines the thresholding value.
Now we have the camera image coming through, we can move to the next step. Remember: our final goal is to take images of doodles on a post-it note and convert it into a musical sequence. So we will need the computer to “see” the drawn lines!
To make it easier, we first simplify the image. The process we use here is called thresholding. For each pixel in the image, we compare its grayscale “brightness” value to a predefined thresholding limit. If it is brighter than the limit, we colour the pixel white, else we colour it black. This way we obtain an image composed of only fully black and fully white pixels.
Just before we apply the threshold filter to the image, we actually introduce a bit of blur. Why? To deal with noise, which is a phenomenon found in photography where stray photons hitting the camera sensor cause nearby pixels to take on different values, where in a perfect world they should be the same colour. If we do not apply this blur, then in the thresholded image we can get specky looking regions of pixels which have values near to the thresholding limit.
Finally for purely aesthetic reasons we remap the black and white pixels to two lovely contrasting pastel shades!
This sketch was my first time using openCV, which is a very widely used computer vision library. I used the javascript distribution which is, in contrast, not widely used at all and I had to make do with little documentation and help on StackExchange! I owe a lot of credit to this sketch, which laid the openCV groundwork for me.