Dec 11, 2020
Intro to ML Final Documentation
I conceived of the idea for my project by deciding to revisit an old sketch I did in Creative Computing last year, and make it more robust with the element of machine learning.
The motivation behind the project was already present at the time I thought to do this sketch. I was after creating a fun and not-so-serious game that addresses a serious issue in the world today. The concept was to highlight the issue of plastic pollution in our oceans, and just how difficult it is to clean it all up, especially with so much microplastics.
The general concept of the game was that each piece of plastic removed was +1 points and every fish “killed” would be -2 points. At the end, the user would get a report of how many individual pieces of plastic were cleaned up.
The next step was to train the Ml5 Handpose model. To do this, I modified Dan Shiffman’s Posenet training sketches, to accomodate the different syntax needed for Handpose.
At first, I decided to eliminate the need for the model to record and Z, or depth, values. This was to help make the training data smaller so it could actually fit into the training sketch. (p5 has a file limit of 5mb). This allowed me to hold my desired hand pose for a second longer for each pose.
Unfortunately that didn’t work too well, and my trained model always leveled out at a pretty high loss value.
Another bug that I kept running into, was that the model kept thinking these two poses, for example, were the same thing. My guess is that it’s likely caused by the same 8–9 points along my palm and at the knuckles being present in both despite the fingers being different.
This resulted in having to get creative and really figure out completely different hand poses for each direction. I ended up needing to differentiate my Left + right, and Up + Down, by using different Z values.
The model seemed to get much more accurate once I added Z values back into the training set as well.
This was the final trained model inside of p5
This was the first time I managed to get the Net cursor to move at all inside the game — it felt like magic when it actually went somewhere.
Here was the final result, with the net being able to move in all directions (albeit rather very inconsistently)
Links to Sketches:
Data Collecting: https://editor.p5js.org/yetiblue/sketches/xqzb99var
Training: https://editor.p5js.org/yetiblue/sketches/xrz37012t
Classifier: https://editor.p5js.org/yetiblue/sketches/PoRAzuH53
