Eye Tracking DLL
Skills: Unity, DLL, C++, C#, OpenCV
Eye Tracking DLL is a project built using OpenCV to create DLL APIs to be used in Unity. With this DLL, programmers are able to locate where the user's eyes fall on the screen.
Idea
The idea behind this project is to create a tool that can help the programmer locate where the user is looking. With this tool, programmers are able to create games that don't require a mouse to control the pointer, because the players can use their eyes instead. As you can imagine, this notion can lead to a lot of amazing possibilities. For example, programmers can create games for people who are not able to use their hands to control the mouse, or free the hands of the players, so that they can use ten fingers on the keyboard to do more complex control.
Program(See the sources code)
In the program, an eye tracking module is used to get the region of interest.
The blue frames the face of the user, and the green frames the eyes.
Then, the colors around the eyes are changed into white, red or green for further calculations.
As shown below, the algorithm is based on a simple notion.
When the user is looking down, the upper part of the eyes will have more green, and if the user is looking right, the right of the eyes will have more green. Therefore, I used quite simple calculations to get the horizontal and vertical values of the position of the players' line of sight as shown below.
Horizontal Value
=
left white pixels of the left eye / (left white pixels of the left eye + left white pixels of the right eye )
+
right white pixels of the right eye / (right white pixels of the right eye + right white pixels of the left eye)
/ 2
Vertical Value = left eye pixels + right eye pixels
Then the position of the line of sight is estimated using the average of a window's past data as shown below.
The window of recent data of the user's line of sight's horizontal and vertical values.
The data is sorted and extreme data is removed before the average is calculated.
The following shows the final formula.
n is the newest Value, and window size is 100.
Demo Games
Now that we have the tool, let me show you two games I made with the DLL's APIs.
Game1: Rescue Game Prototype
In this game, I used eye tracking and voice recognizing to create a prototype which I hope can be a stepping stone for creating games for physically challenged people in the future. In this game, the player can enjoy the game without using their hands.
Gameplay & Background Story
In this game, the player will help a girl get out of a magical maze covered by mist. By using their eyes to see through the mist, the player then uses their voice to tell the girl where to head.
Game2: PewBombDie with eye aiming
PewBomeDie is a multiplayer shooting game I made. I think it will be great to let the player use eyes instead of the mouse to aim so that the player can use the spared hand to control the keyboard.
Gameplay
In the game, the players will try to use different kinds of weapons and skills to kill other players and try to be the last one standing; for more details, see PewBomeDied. The following is the video of me using eye tracking to aim while playing with my friends who used the mouse to aim.
Conclusion
By implementing the eye tracking APIs in two games, I demonstrated the new possibilities of game design with the help of eye tracking techs. However, as you can see in the videos that the accuracy of the APIs not that satisfying. It's because the webcam that I used has a lower resolution, so the regions of the eyes contain very few pixels; hence, every pixel weighs too much and can lead to great miscalculation.