This article has moved to:
http://opencv-code.com/Real_Time_Eye_Tracking_and_Blink_Detection
This article explains how to implement real time eye tracking and blink detection with OpenCV. Source code is available for download.
This project is an implementation of the algorithm described in the paper: Real Time Eye Tracking and Blink Detection with USB Cameras (download pdf) by Michael Chau and Margrit Betke.
The paper described an algorithm to detect user's eye blinks and eye tracking that sounds promising, so I tried to implementing it with some small modifications. And I shamelessly use the paper's title for my own page.
This system is the enhancement of my previous Eye Tracking system, where this system automatically locate the user's eye by detecting eye blinks. Motion analysis techniques are used in this stage, followed by online creation of the open eye template. The open eye template is used to locate the user's eye in the subsequent frames with template matching. Blink detection is performed using motion analysis techniques.
Since the operation requires extensive amount of computation, the search region is restricted in a small search window around the user's eye. This method will drastically reduces the computation needed thus making the system running smoothly in real time.
In this stage, the system will try to locate the eyes by analyzing the blinking of the user. Given the current grayscaled frame gray
and the previously saved frame prev
, we obtain the difference image diff
. The difference image then thresholded, resulting a binary image showing the regions of movement that occured between two frames.
Listing 1: Motion analysis to detect eye blinks
The remove noise and produce fewer and larger connected components, a 3x3 star-shaped convolution kernel is passed over the binary image in an Opening Morphological operation.
Listing 2: Opening Morphological operation
Connected component labeling is applied next to obtain the number of connected components in the difference image.
Listing 3: Connected component labeling
The function above will return the connected components in comp
, as well as the number of connected components nc
.
At this point, we have to determine whether the components are eye pair or not. We'll use experimentally derived heuristics for this, based on the width, height, vertical distance, and horizontal distance of the components. To make things simple, we only proceed if the number of the connected components is 2.
Here are the rules applied to determine whether connected components are eye pair:
If the components successfully pass the filter above, the system will continue with the online template creation.
After the connected components passed the heuristics filter above, the system will obtain the boundaries of the first connected component, rect_eye
. It will be used to extract a portion of the current frame as the eye template.
Listing 4: Online template creation
Note that we set some delay before creating the template. That's because what we need is an open eye template. Since the user's eyes are still closed at the heuristics filtering above, we'll wait a moment for the user to open his eyes.
Having the eye template and live video feed from camera, the system will try to locate the user's eye in the subsequent frames using template matching. The searching is limited in a small search window since searching the whole image will use extensive amount of CPU resources.
Listing 5: Locating the eye in subsequent frames
The location of the best matches is available in minloc
. It will be used to draw a rectangle in the displayed frame to label the object being tracked.
Not only locating the user's eye, the system also detect user's blinks. In the algorithm from the paper above, they detect eye blinks by analyzing the correlation score from the eye tracking stage. In theory, when the user blinks the similarity to the open eye template decreases.
While it is true in most cases, I've found that it is only reliable if the user doesn't make any significant head movements. If the user move his head, the correlation score also decreases even if the user doesn't blink.
In this system I use motion analysis to detect eye blinks, just like the very first initilization stage above. Only this time the detection is limited in a small search window, the same window that is used to locating the user's eye.
Listing 6: Blink detection with motion analysis
cvFindContours
will return the connected components in comp
, and the number of connected components nc
. To determine whether a motion is eye blink or not, we apply several rules for the connected component:
Note that we require only 1 connected component, while normally user blink will yielding 2 connected components. That's because we perform the motion analysis in a small search window, where the window fits only for 1 eye.
Below are some screenshots of the system.
Notice the second image, the outer rectangle is the search window and the inner rectangle is the location of the object being tracked. In this stage, eye blinking yielding only 1 connected component since the searching is restricted in the search window. Also note the text 'blink!' displayed in the third image. The text is displayed when the system detect user's blinks.
Learning OpenCV: Computer Vision with the OpenCV Library
By: Gary Bradski, Adrian Kaehler
This book is the "de facto" OpenCV User's Manual. It provides a practical, pragmatic, accessible book on computer vision, with algorithmic explanation and concrete example code snippets. Written by the creators of OpenCV, no doubt you should obtain a copy.
Machine Vision: Theory, Algorithms, Practicalities (Signal Processing and its Applications)
By: E. R. Davies
This book provides a solid and concrete foundation to computer vision from engineering point of view. Use Learning OpenCV from Gary Bradsky to learn how to use OpenCV, and use this book to understand how OpenCV works behind the screen.
C Programming Language (2nd Edition)
By: Brian W. Kernighan
For you who new to C programming, this is a must-have book for you. Written by the C creator himself, this book is concise and powerful, just like C itself.
Nash on Jul 16, 2009:
k09 on Jul 20, 2009:
Nash on Jul 20, 2009:
#define FRAME_WIDTH 240
#define FRAME_HEIGHT 180
k09 on Jul 22, 2009:
Nash on Jul 23, 2009:
naraba on Aug 20, 2009:
Nash on Aug 20, 2009:
naraba on Aug 21, 2009:
VR on Sep 18, 2009:
Nash on Sep 18, 2009:
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, FRAME_WIDTH);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT);
VR on Sep 21, 2009:
DiegoZ on Sep 23, 2009:
Nash on Sep 23, 2009:
Indigo on Sep 28, 2009:
Nash on Sep 28, 2009:
AB on Oct 7, 2009:
if (!found || key == 'r' || _________)
stage = STAGE_INIT;
ronhab on Nov 18, 2009:
Abdul Wahab on Feb 28, 2010:
Barbara on Mar 4, 2010:
sanchit gupta on Apr 5, 2010:
Sergio on Apr 6, 2010:
Nash on Apr 6, 2010:
Gus on Apr 8, 2010:
Sergio on Apr 12, 2010:
J on May 4, 2010:
Aymeric on May 5, 2010:
lucky0403 on May 31, 2010:
Ozw on Jun 4, 2010:
lyuba on Jun 6, 2010:
lyuba on Jun 6, 2010:
Nash on Jun 6, 2010:
lyuba on Jun 6, 2010:
Ozw on Jun 9, 2010:
lek on Jun 22, 2010:
kiko on Jul 15, 2010:
Frodo on Aug 17, 2010:
Frodo on Aug 17, 2010:
kiko on Sep 2, 2010:
sancelot on Sep 9, 2010:
Steve on Oct 16, 2010:
yol on Nov 3, 2010:
venkatesh on Nov 6, 2010:
tayo ejidokun on Nov 22, 2010:
Fayas on Nov 25, 2010:
Nash on Nov 25, 2010:
sreevathsan on Mar 8, 2011:
adrian on Apr 1, 2011:
guillermo on Apr 14, 2011:
Ram on Apr 14, 2011:
guillermo on Apr 14, 2011:
quakerr on Jun 27, 2011:
quakerr on Jun 27, 2011:
sabih on Jul 20, 2011:
Mohammed on Jul 22, 2011:
ICQ | 489571630 |
Skype | dede_bl4ckheart |
Yahoo | dede_bl4ckheart |
nashruddin.amin |
k09 on Jul 16, 2009: