This article has moved to:
http://opencv-code.com/Real_Time_Eye_Tracking_and_Blink_Detection

Real Time Eye Tracking and Blink Detection

Jul 16, 2009 | Tags: OpenCV, Projects | del.icio.us del.icio.us | digg Digg

This article explains how to implement real time eye tracking and blink detection with OpenCV. Source code is available for download.

This project is an implementation of the algorithm described in the paper: Real Time Eye Tracking and Blink Detection with USB Cameras (download pdf) by Michael Chau and Margrit Betke.

The paper described an algorithm to detect user's eye blinks and eye tracking that sounds promising, so I tried to implementing it with some small modifications. And I shamelessly use the paper's title for my own page.

Table of Contents:

  1. Introduction
  2. Initialization
  3. Online Template Creation
  4. Eye Tracking
  5. Blink Detection
  6. Image Gallery

1. Introduction

This system is the enhancement of my previous Eye Tracking system, where this system automatically locate the user's eye by detecting eye blinks. Motion analysis techniques are used in this stage, followed by online creation of the open eye template. The open eye template is used to locate the user's eye in the subsequent frames with template matching. Blink detection is performed using motion analysis techniques.

Since the operation requires extensive amount of computation, the search region is restricted in a small search window around the user's eye. This method will drastically reduces the computation needed thus making the system running smoothly in real time.

2. Initialization

In this stage, the system will try to locate the eyes by analyzing the blinking of the user. Given the current grayscaled frame gray and the previously saved frame prev, we obtain the difference image diff. The difference image then thresholded, resulting a binary image showing the regions of movement that occured between two frames.

Listing 1: Motion analysis to detect eye blinks

cvSub(gray, prev, diff, NULL);
cvThreshold(diff, diff, 5, 255, CV_THRESH_BINARY);

The remove noise and produce fewer and larger connected components, a 3x3 star-shaped convolution kernel is passed over the binary image in an Opening Morphological operation.

Listing 2: Opening Morphological operation

IplConvKernel* kernel;

kernel = cvCreateStructuringElementEx(3, 3, 1, 1, CV_SHAPE_CROSS, NULL);
cvMorphologyEx(diff, diff, NULL, kernel, CV_MOP_OPEN, 1);

Connected component labeling is applied next to obtain the number of connected components in the difference image.

Listing 3: Connected component labeling

CvSeq* comp;

int nc = cvFindContours(
    diff,                   /* the difference image */
    storage,                /* created with cvCreateMemStorage() */
    &comp,                  /* output: connected components */
    sizeof(CvContour),
    CV_RETR_CCOMP,
    CV_CHAIN_APPROX_SIMPLE,
    cvPoint(0,0)
);

The function above will return the connected components in comp, as well as the number of connected components nc.

At this point, we have to determine whether the components are eye pair or not. We'll use experimentally derived heuristics for this, based on the width, height, vertical distance, and horizontal distance of the components. To make things simple, we only proceed if the number of the connected components is 2.

Here are the rules applied to determine whether connected components are eye pair:

  1. The width of the components are about the same.
  2. The height of the components are about the same.
  3. Vertical distance is small.
  4. Reasonable horizontal distance, based on the components' width.

If the components successfully pass the filter above, the system will continue with the online template creation.

3. Online Template Creation

After the connected components passed the heuristics filter above, the system will obtain the boundaries of the first connected component, rect_eye. It will be used to extract a portion of the current frame as the eye template.

Listing 4: Online template creation

cvWaitKey(250);

cvSetImageROI(gray, rect_eye);
cvCopy(gray, tpl, NULL);
cvResetImageROI(gray);

Note that we set some delay before creating the template. That's because what we need is an open eye template. Since the user's eyes are still closed at the heuristics filtering above, we'll wait a moment for the user to open his eyes.

4. Eye Tracking

Having the eye template and live video feed from camera, the system will try to locate the user's eye in the subsequent frames using template matching. The searching is limited in a small search window since searching the whole image will use extensive amount of CPU resources.

Listing 5: Locating the eye in subsequent frames

/* get the centroid of eye */
point = cvPoint(
    rect_eye.x + rect_eye.width / 2,
    rect_eye.y + rect_eye.height / 2
);

/* setup search window */
window = cvRect(
    point.x - WIN_WIDTH / 2,
    point.y - WIN_HEIGHT / 2,
    WIN_WIDTH,
    WIN_HEIGHT
);

/* locate the eye with template matching */
cvSetImageROI(gray, window);
cvMatchTemplate(gray, tpl, res, CV_TM_SQDIFF_NORMED);
cvMinMaxLoc(res, &minval, &maxval, &minloc, &maxloc, 0);
cvResetImageROI(gray);

The location of the best matches is available in minloc. It will be used to draw a rectangle in the displayed frame to label the object being tracked.

5. Blink Detection

Not only locating the user's eye, the system also detect user's blinks. In the algorithm from the paper above, they detect eye blinks by analyzing the correlation score from the eye tracking stage. In theory, when the user blinks the similarity to the open eye template decreases.

While it is true in most cases, I've found that it is only reliable if the user doesn't make any significant head movements. If the user move his head, the correlation score also decreases even if the user doesn't blink.

In this system I use motion analysis to detect eye blinks, just like the very first initilization stage above. Only this time the detection is limited in a small search window, the same window that is used to locating the user's eye.

Listing 6: Blink detection with motion analysis

/* motion analysis
   cvSetImageROI has been applied to the images below */

cvSub(gray, prev, diff, NULL);
cvThreshold(diff, diff, 5, 255, CV_THRESH_BINARY);
cvMorphologyEx(diff, diff, NULL, kernel, CV_MOP_OPEN, 1);

/* detect eye blink */
nc = cvFindContours(diff, storage, &comp, sizeof(CvContour),
                    CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE, cvPoint(0,0));

cvFindContours will return the connected components in comp, and the number of connected components nc. To determine whether a motion is eye blink or not, we apply several rules for the connected component:

  • There is only 1 connected component.
  • The component is located at the centroid of user's eye.

Note that we require only 1 connected component, while normally user blink will yielding 2 connected components. That's because we perform the motion analysis in a small search window, where the window fits only for 1 eye.

6. Image Gallery

Below are some screenshots of the system.

Screenshots: (a) The difference image when the user blink in the initialization stage. (b) The user blink in the tracking stage. (c) The video displayed to user. The red rectangle is the search window and the green rectangle is the object being tracked.
Fig 1. Screenshots: (a) The difference image when the user blink in the initialization stage. (b) The user blink in the tracking stage. (c) The video displayed to user. The red rectangle is the search window and the green rectangle is the object being tracked.

Notice the second image, the outer rectangle is the search window and the inner rectangle is the location of the object being tracked. In this stage, eye blinking yielding only 1 connected component since the searching is restricted in the search window. Also note the text 'blink!' displayed in the third image. The text is displayed when the system detect user's blinks.

Related Articles

Recommended Books

The Downloads

56 Comments

k09 on Jul 16, 2009:

Very impressive. Sorry because I had a trouble about Internet connection so I couldn't update your information. I built a same project but it is very simple level (selecting template by mouse) so your project is very useful to me. To learn from you, would you mind if I download your sourcodes and use some parts for my project?

Nash on Jul 16, 2009:

No problem. It is GPL-ed.

k09 on Jul 20, 2009:

When the frame size is large (for example 360x270), more connected components are produced (more noise) so we can not perform eye tracking. To solve this problem, I think there is a method to examine all connected components and decide whether a component is eye or not.

Nash on Jul 20, 2009:

Yes, bigger frames will produce more connected components and the system would need a more complex filter to obtain user's eye pair. That's why I used small frames from the first place:

#define FRAME_WIDTH  240
#define FRAME_HEIGHT 180


Smaller frames yield less connected components, so the program can easily decide whether the components are eye pair or not. But be aware that the code above is intended to make things simpler and easier for implementing a bigger application. Here's the idea:

1. Obtain a frame from camera (say 800x600 pixels).

2. Resize the frame to a smaller image (say 160x120).

3. Use the 160x120 image for eye detection, eye tracking, and blink detection.

4. Obtain the location of the user's eye in the 800x600 frame, using some sort of calculation.

5. Display the 800x600 frame to the user.

With this method, we can reduce significant CPU workload compared with using the 800x600 frame for all of the steps. Also the code would be a lot simpler just like this one.

See, above is just a small piece of code from a more bigger and complex application.

k09 on Jul 22, 2009:

Thanks for your advice. I've implemented it in my project. By the way, why didn't you use the correlation coefficient matching methods(method = CV_TM_CCOEFF) to determine the user's eyes in the tracking stage. I think it is more accurate. For ex:

// threshold to determine tracking eyes
#define TE_THRESHOLD    0.4

// threshold for open eyes' template
#define OE_THRESHOLD    0.85

// threshold to determine blinking
#define UB_THRESHOLD    0.8
#define LB_THRESHOLD    0.55

cvMatchTemplate(img, tpl, tm, CV_TM_CCOEFF_NORMED);
cvMinMaxLoc(tm, &minval, &maxval, &minloc, &maxloc, 0);

if (maxval < TE_THRESHOLD)
    return 0;
   
// return the search window
*window = win;

// return eye location
*eye = cvRect(
    win.x + maxloc.x,
    win.y + maxloc.y,
    TPL_WIDTH,
    TPL_HEIGHT
);

// eye closed
if ((maxval > LB_THRESHOLD) && (maxval < UB_THRESHOLD))
    return 2;

// eye opened
if (maxval > OE_THRESHOLD)
    return 1;

Nash on Jul 23, 2009:

It looks good. I'll give it a try.

naraba on Aug 20, 2009:

I know nothing about computer vision (and also I'm poor at English ), so I may be wrong.

To extend the search region, I think the face tracking might be useful for reducing the total amount of computation required. In this case, there are two tracking stage; i) the face tracking stage searches the user's face within the whole search region to focus the most likely area, and then ii) the eye tracking stage searches the user's eye only within the focused area. In this method, I think the system can also deal with the head-movement.

How effective do you think this method is? Or rather, even worse, it just increases the computation?

Nash on Aug 20, 2009:

I think the algorithm you described is pretty good. It is very similar with my eye detection code. Its only that I was trying to implement the algorithm from the paper of Michael Chau.

naraba on Aug 21, 2009:

Oh, very sorry for not finding you've already done exactly what I described...

VR on Sep 18, 2009:

Good stuff... I ran the sample code but failed to get good results. I am hoping you could point me in the right direction. The issue I have is that the code does not get into Tracking Stage because nc in is_eye_pair is never equal to 2. I have made no changes to your code except trying out K09's recommendation for accuracy. As I am not even getting to close to nc==2, I wonder if I am missing something... I have compiled this on a Ubuntu system and am using a Dynex camera. Thanks in advance.

Nash on Sep 18, 2009:

Actually, it is normal. Connected components _usually_ more than 2, like described in the paper above. So we must deploy complex heuristics to determine the user's eye pair.

The code above only continue if nc==2 just to make the heuristics simpler. Therefore I resize the frames to smaller size to reduce noises. I guess the frames taken from your camera bigger than 240x180? If that is the case, most likely it is because the OpenCV version you're using is 1.0. These code won't work:

cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, FRAME_WIDTH);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT);


Try to upgrade to OpenCV-1.1pre1 and see what happen.

VR on Sep 21, 2009:

Thanks Nash. That was it... as I had said earlier... excellent work. Did you ever look into gazing aspect of the eye tracking? There is some work (open source) that has already been done in this area and could be a good addition to this code.

DiegoZ on Sep 23, 2009:

Hi, I'd like to know if somebody knows if is it possible to control the mouse movement outside the opencv's frame? What exactly I want is to control the mouse over the windows os. Is it possible?

Nash on Sep 23, 2009:

Maybe this will help:
http://www.codeguru.com/forum/showthread.php?t=377394

Indigo on Sep 28, 2009:

I don't seem to be able to get the file to download correctly.... or how do i open a .c file?

Nash on Sep 28, 2009:

Man, you have so many homework to do. OpenCV wiki is a good start.

For you non-programmer readers who want to try my code above, I've provided a pre-compiled version for Windows. Get it at the download section. Happy testing.

AB on Oct 7, 2009:

Hi,

Great work! I was trying to incorporate k09's eye tracking algorithm suggestions into this program, but was unsure where to add that code and what to remove. Any suggestions?

Also, I am trying to have it so that when the search window does not move for some amount of time, that it reinitializes (i.e. stage = STAGE_INIT). I assume I would need to put something in the line:

if (!found || key == 'r' || _________)
  stage = STAGE_INIT;


but I'm not sure how to fill in the blank. Any help would be greatly appreciated.

Thank you!!

ronhab on Nov 18, 2009:

Hi,

Great code! (I also read and try to implement the same algorithm from the same article as part of my graduate project).

One thing I can't understand:
when you find a "eye pair" during the INIT stage, you do:
delay_frames(5);

before copying the template. This suppose to let the user open his eye in order to get open eye template. But, looking at your it seems like delay_frames only update the "frame" image and not "gray" image - so after the delay the template will still be taken from the same frame - with the user eyes close.

Am I wrong?

Abdul Wahab on Feb 28, 2010:

Great work buddy.. its really impressive.. im also tring to do similar eye tracking work but im having lot4s of problems.. i hope your code will be useful..

Thanks alot..

Barbara on Mar 4, 2010:

Hi,
It is really a very cool project. I had run your code. I modified my frame to 176*144. I think is small enough, is it? And I also use the opencv 1.1pre. But I still cannot get 2 connected component. I get about 50 connected components. I don't know whether light and camera solution can affect the number of connected components?
What else can I do to reduce the number of components?

Now I just iterate and try every association between those 50 components. But in that case, the result of is_eye_pair check is not accurate.

sanchit gupta on Apr 5, 2010:

Hey I am interested in doing something similar to this . i was perhaps thinking of doing something like motion detection using opencv as my final yr proj. Do u have some code on motion detection or tracking using predefined object. i wish to build a virtual mouse tracking system using motion tracking.
if possible pls mail me at sanchit30@gmail.com

Sergio on Apr 6, 2010:

Hello,
First of all, I'd like to thank you for publishing your work.
I compiled your code under GNU/Linux Operating System. It compiles correctly except for the following warning:

In function ‘exit_nicely’:
: warning: format not a string literal and no format arguments

The thing is, when I run the program, it prompts a window showing my webcam's capture but there is no eye tracking at all. After several seconds the program prompts a second window showing the 'diff' image. Again, there is no rectangles whatsoever.
I hope you can give me some feedback about this.

Nash on Apr 6, 2010:

blink your eyes so the system can locate them.

Gus on Apr 8, 2010:

@sergio.

I've had the same problem: compiling was Ok, the program runs, the diff window showed up, but no rectangle. The problem was that the ambient light was blinking. Get a good ambient lighting and try it again. Don't forget to tell us about your results!

Sergio on Apr 12, 2010:

Hello guys,
Apparently the problem is that the webcam capture is not being resized, as if these commands don't have any effect on the capture:

cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_WIDTH, FRAME_WIDTH);
cvSetCaptureProperty(capture, CV_CAP_PROP_FRAME_HEIGHT, FRAME_HEIGHT);

J on May 4, 2010:

getting the Cannot initialize camera! error. its nothing about your code, but about my iSight and the cvCaptureFromCAM(). does anybody have experience with isight?
Joe

Aymeric on May 5, 2010:

Hello,

First of all, thank you very much for publishing this very useful code.
To determine the user's eyes in the tracking stage, I'd like to implement the correlation coefficient matching method that was described by k09.
Did anyone succeed in implementing it ?

Because I've inserted the code in"locate_eye" method, instead of template matching method, but it doesn't work. As soon as one eye is being tracked, I've got a Runtime Error indicating me a problem with "cvCvtColor" method.

Thanks for your answer.

lucky0403 on May 31, 2010:

dear Nash can you give me some suggests about the detection of mouth state(closed or open)?

Ozw on Jun 4, 2010:

Thanks for the code, great work. any chance to get a new version that can be compiled with openCV 2.1 (last one) on vs2010 or 2008?

lyuba on Jun 6, 2010:

2 Nash
First of all, I would like to thank you for your projects and blog. They are great, understandable, and that's really cool that you help people

Secondly would like to ask, why did you choose C, not C++ version of the OpenCV. Is it because of the speed?

OpenCV 2.1. C++ seem to be pretty usable: http://opencv.willowgarage.com/documentation/cpp/introduction.html

I am now trying to decide which one to useAppreciate you tip.

2 Ozw
Can't tell you a thing about VS, but the code is compiled great with the new OpenCV 2.1. with gcc. OpenCV seems to have reliable backward compatibility. Good luck!

lyuba on Jun 6, 2010:

It compiles fine, but blinks are not unfortunately found. What may be the reason? I try blink like crazy ))
By the way, it shows only two screens: video and diff. In the article it looks like showing three of them.
Thanks!

Nash on Jun 6, 2010:

I prefer C because it simply works. The program displays 2 windows. The article shows the 'diff' window at two different stage. If the blink isn't detected, check if the 'diff' window shows 2 connected components like the screenshots above.

lyuba on Jun 6, 2010:

Nash,

thank you for your answer. So C++ doesn't work and you had problems with it?

I have a window, which displays the difference. However no blinks are detected. I also tried to vary the distance from the screen - no result.

Ozw on Jun 9, 2010:

Thanks lyuba and Nash for quick response. It appears that VC++ expects all declarations to at the first rows of function scope. I've separated between declaration and assignment and now I finally compiled it successfully.

lek on Jun 22, 2010:

Hi Nash,

I am started succesfully with a couple of OpenVC 2.0 demos using VisualC++ 2008. I have tried your binaries with my system and everything is ok (great work!).

The problem arises when I try to put your blink.c to work with my system. It compiles ** but doesn't run. I get the message "The procedure entry point cvCreateCameraCapture could not be located in the dynamic link library libcv200.dll". What could be wrong?

** to compile I renamed your blink.c to blink.cpp and added a
#include "stdafx.h" header
I got the following warning LNK4078: multiple '.text' sections found with different attributes (E0300020) libcv200.dll.a OpenCVMyTests

kiko on Jul 15, 2010:

Hi nash ! I am also using Visual 2008 C++ but i did not understand the changes Ozw made to make it work !!!! I only get the command line and and a small screen with gray lines ...my camera works with opencv and you your other project "eye tracking" works perfectly.

Frodo on Aug 17, 2010:

Good work nash!!!!!
But i guess the accuracy of tracking and blinking decreases with K09's changes in locate_eye function.

First of all it loses the tracking and gets re-initialized and eye blinking will not be detected compared to the nash's code.....

Code can be customized if the blinking is not detected or delay in blinking also which might be helpful for specific apps....(this is in fact easy)!

Frodo on Aug 17, 2010:

Good work nash!!!!!!!

Accuracy of tracking and blink reduces with k09's piece of code if implemented compared to your(nash) code.

I am trying to make this more robust by combining with face and eye detect haar cascade, i am successful to some extent but again the accuracy of blink detection is reducing. Any ideas about this??

Algorithm that i am doing:
1)Detect face
2)set the ROI to eyes position based on the co-ordinates got from face detection.
3)This will be the input image(ROI image) to your part of the code rather the whole image.

Why i did this is any movement in the background reduces the accuracy of tracking and blink detection.
Let me know your thoughts on this......

kiko on Sep 2, 2010:

it is me again, I made it work in Visual 2008 C++ i wont say the changes i made,witch are a lot but I suggest that you start with simpler programs with opencv first to get familiar with how it works and then try to understand this one !!!it is a little puzzle!!you will see that the code starts of with something that is not necessary for it to run....

sancelot on Sep 9, 2010:

Hi, nice job, but like some people I can not achieve to have blink detection, I have only diff and camera view, I am using ubuntu lucid

Steve on Oct 16, 2010:

Hi Nash!

Good job on the code! I've compiled it using Visual Studio 2008 C++ express and it works great on my computer! However, when I build the code and send it to a friend who doesn't have VS2008 or openCV installed, he can't open the application. I've already seen that you've added a prebuilt version for windows. When I try to add the same .dll files to a zip-file however, I still can't execute the program on other computers than mine. Any idea what might be the problem?

Thanks in advance,

Steve

yol on Nov 3, 2010:

The blink detection doesnt work that good if your capture resolution is to big.
When using ubuntu i found that the cmake step of installing opencv doesnt include the v4l driver(or somthing like that) if the libv4l-dev package isnt installed.
You will notice that you cant set a resolution if so.

sudo apt-get instal libv4l-dev

And rerun the cmake and make install part of opencv installation and the resolution will work.

venkatesh on Nov 6, 2010:

hii nash nd evryone above....
am workin on a project to detect DRIVER'S DROWSINESS. For that i choosen EYE-BLINK patterns are the best criteria. ur code s very helpful and gave me a enthusiastic start.
have some issues
1.camera specifications.
2.coding platform.

tayo ejidokun on Nov 22, 2010:

hii nash .....good job you done so far here am a bit inspired and want to start some thing on my own but i have a problem of installing openCV to work in my visual studio enviroment and i think its am using windows visa..............i will be grateful if am given useful tip s on how to about it.

Nash on Nov 23, 2010:

Check out this tutorial.

Fayas on Nov 25, 2010:

Hello,
Do you have any openCV code for detecting the smile, opened mouth or detecting the teeth region.

Nash on Nov 25, 2010:

Sorry I don't have the code. I'm still learning the subject.

sreevathsan on Mar 8, 2011:

sir,
Your Tutorials were excellent.Thanks for that.
i m able to detect the eye region using Adaboost Algorithm. Now i want to classify whether the person is awake or asleep.
My mini project is on sleep detection and i want to detect the sleeping people.
Please give me some suggestions.
thanks in advance.

adrian on Apr 1, 2011:

great work with that code!!!! I only have a question for you... Is there a way to modify your code to make it detect when someone falls asleep?? for example I want to do a program that detects when my kid closes his eyes for more than 5 sec... I am new to OpenCV so I was thinking of doing a function that looks for white or closed to white pixels on your search rectangle and if they disappear for more than 5 secs then my kid is falling asleep.. Do you think that it will work?? Perhaps could you help me??

Thank you very much for your time!

guillermo on Apr 14, 2011:

Hello nash, first of all, thanks for your work, its been very helpfull for me, and i'm learnt so much. Im working on your projest, but im having problems with the memory, because your code spend it more and more, and i dont find where. I dont know if the problem is the algorithm or the implementetion. I've read the code so many times and it seems that all IplImages are released... Can somebody help me?

Ram on Apr 14, 2011:

Hey ,
we just compiled your blink.c code.but we were able to get only the gray scale video and could not get the eye blink detection part.could you just help us out on this and let us know where we are missing out on at the earliest.

guillermo on Apr 14, 2011:

hi again. I think the leak is in the structure CvCapture:
when you call: frame = cvQueryFrame(capture);more memory is needed and allocated in the structure capture, but i dont know how deallocate it.

quakerr on Jun 27, 2011:

Hi venkatesh,

i'm working on a similar proj, but instead of eye blink pattern, i use head pose estimation and eye gaze vectors to find out if the driver is concentrating on the road. I would like to know, how your project has shaped up so far. contact me at quakerr90@gmail.com

quakerr on Jun 27, 2011:

Hi venkatesh,

i'm working on a similar proj, but instead of eye blink pattern, i use head pose estimation and eye gaze vectors to find out if the driver is concentrating on the road. I would like to know, how your project has shaped up so far. contact me at quakerr90@gmail.com

sabih on Jul 20, 2011:

great work Nash.

i want to use ur code to perform eye tracking of AR Drone camera.

the AR Drone source codes are provided and we can write our own code to determine its flight. etc.
can you tell me how to integrate ur code with AR drone code so that i can achieve this??

Mohammed on Jul 22, 2011:

Hi Nash
Thank you for publishing your great work
I have a question, and will be very grateful if you answer:
I want to detect the position of pupil in the green rectangle. How can I do this?
Thanks for your help

Comment is closed.