Sparkles on Fingertips
Skills: Unity, OpenCV, Mediapipe, ML.NET, Python, C#
This project is a prototype of a 2D platform video game. In the game, I used OpenCV, Mediapipe and ML.NET to create the function of hand/finger recognition and symbol recognition.
Game Idea
We all have played RPG games in which the player can cast cool spells, such as fire balls, to attack enemies. However, to me, it does not feel like really drawing and casting spells by just pressing buttons. Therefore, by allowing the player to use their hands instead of buttons to conjure or cast the spells, I aim to let the player really feel like they are indeed the character in the game. At first, my intention was to let the player cast spells by drawing a combination of symbols. For example, if symbol A represents big and symbol B represents fire ball, then the player can combine A and B to create a big fire ball. However, in order for the program to distinguish that many symbols, a lot of training data has to be fed to the training model. Since I made my own training data, it would take me too much time to do so, so I changed the plan and made each symbol represent a spell.
Game Story
The idea of the protagonist came from Douxie, who is a character in a TV show called Tales of Arcadia. The protagonist of the game is a witch who has a cool device on her wrist that can contain spells she draws. When needed, she can swipe the interface of the device to choose a spell she wants to use and then cast it.
Gameplay
In the game, using their index finger to draw in front of the webcam, the player can draw spells, which will be stored and then used to attack enemies later.
Then, the player can choose spells by swiping both index and middle fingers in the air.
Finally, the player can cast the spell and attack the enemies by holding and then releasing their hand.
There are three kind of spells in the game: Fire Spell, a fire ball that is shot from the front of the avatar, and then cause damage to enemies it hit, Lightning Spell: lightning that strike the ground from the position of the player’s hand, and then deal damage to enemies it hits, and Tornado Spell: a tornado that are released from the position of the player’s hand, and moves forward while pushing and hurt enemies it comes across with.
Fire Ball
Lightning
Tornado
Programming
In the project, the program first has to be able to detect the player's hands and fingers in Unity. The detection was fulfilled by using Google's Mediapipe, which is a very useful but not yet completed API made with OpenCV, which can be used for computer vision. However, the problem is that Mediapipe doesn't support C# and Unity, so I had to embed a whole Python environment into the Unity project in order to use Mediapipe in my project. After that, ML.MET and the training data are used to classify the symbols that the player draws.
Conclusion
By combining computer vision and machine learning, I successfully implemented the gameplay I desired, which is to allow the player to really use hands instead of buttons and mice to play the game. I believe by using hands instead of buttons to cast spells, the player will easily immerse into the game.
Download and Play
Platform requirements
1. Window x64 system
2. A webcam
Installation
Step 1: Download and unzip the game folder. P.S. The game does not function if the directory it is placed in contains special characters, such as Chinese.
Step 2: Make sure the webcam works and is not occupied by other programs.
Step 3: It works better if the webcam is aiming slightly right so that it can better capture your right hand when you raise it in front of the computer.
Step 4: In the game folder, open the Sparkles on Fingertips.exe file to play.