2009 NFL Season - Passer Ratings

We now have data for the entire 2009 season, and I can calculate the best and worst performers for the season, using my standard measure, CMI - or more appropriately, Standard Deviations from mean CMI.  CMI is easily calculated, as it is simply [pass completion percentage] - 3 * [interception percentage].  I can calculate that at any point during the season.  In order to calculate the standard deviations from the mean CMI, I like to wait until the season is over (actually, even this can be calculated at any point during the season, but it's a lot simpler for me to just do it at the end of the season - perhaps next year, I can do this after every week during the regular NFL season).   For those of you familiar with statistics, the measure of standard deviation from the mean is also called a z-score (or z-value).  And, furthermore, z-score can be very easily converted to a percentile rank (assuming that the population is normally distributed.  In a later post, I will show that z scores for CMTI or NFL passer rating over the years is distributed normally).  The beautiful thing about a percentile rank is that it is always a number between 0 and 100.  What's not to like about that!

A quick couple of notes here regarding CMI, and many of the calculations that I will be using from here on out. While I will go through the exercise of creating arduous, complex and sometimes arbitrary (and even unnecessary) calculations, I will strive for simplicity whenever possible when I present my ideas, as I believe they are more likely to be accepted.   The use of percentile ranks is an example of this.

Last year, when I first developed CMI, and especially as I looked back in time, I had to figure out how many players to use each year to determine the number of qualifiers.  I went through some elaborate mechanisms to calculate the number of passes attempted during the season (per team per game), and some qualifying standard.  As you are aware, the NFL uses 14 passes per game as the standard.  And yes, while it is simple to use, I am not a big fan of that standard since the average # of passes attempted per game has changed (increased) over time, as you can see from the table below.

The standard today should be closer to 16.

Last year, I arbitrarily assigned a factor close to 50% (with adjustments for the earlier years) to the total # of passes attempted by each team per game to determine the # of qualifiers.  I realized this year, that if I simply used the # of teams in the league as the cutoff for determining the # of qualifiers, I get fairly close to my original method.  And because it is simple, easy to explain, does a better job (in my opinion, anyway) than the current NFL standard, I will use that as the cutoff.  The table below shows the # of qualifiers by season using the NFL standard, my original standard, and my current standard.

As you can see, the # of qualifiers, regardless of which method you select is fairly close in each year, with the exceptions being the earliest years, where the NFL method simply excludes too many passers (in my opinion anyway).  Also note that the the AAFC is excluded by the NFL altogether, while the AFL and NFL have been combined in the years 1960-1969 by the NFL.  This doesn't make sense to me, since each league played a different # of games (and had a different # of teams).  Regardless, if you combine the AFL and NFL results from my method, you get close to the #s used by the NFL.  While I think that my method works, I also recognize that this standard may not be the most appropriate for future NFL seasons.

OK, back to the passer ratings.

Because I will discuss other passer rating systems/calculations that are related to CMI in later posts, I will take this moment to re-classify CMI as CMTI (acronym stands for Completions Minus Three times Interceptions).  The definition itself hasn't changed - just the acronym.

The table below shows each of the qualifying quarterbacks with their 2009 season's statistics, their NFL passer rating, the percentile rank of their 2009 NFL passer rating (in other words, the converted z-scores), their 2009 rank in terms of NFL passer rating, their 2008 NFL passer rating, the percentile rank of their 2008 NFL passer rating, their 2008 NFL passer rating rank, their 2009 CMTI, 2009 CMTI percentile rank, their 2008 CMTI and 2008 CMTI percentile rank, and their 2008 CMTI ranking, and finally, their average 2-year CMTI percentile rank (again a # between 0 and 100), together with that ranking.

I don't think there's any question as to who the elite quarterbacks are.  Also notice the struggles of first-year quarterbacks in each year.

Happy New Year!

Hi Everyone.

Happy New Year to you all.

It's been awhile since I've posted.  The reasons are many, so I shall not go into them.  Regardless, I am back posting.

I have been able to do some additional research in the meantime.  I have added data from the 1932 - 1939 NFL seasons to my database.  I have also added the 1946-1949 AAFC seasons to the database as well as the 1960-1969 AFL seasons.  And of course, updated for the just concluded 2009 NFL season.

I think that adding the data from the AAFC in the late 1940's as well as the AFL during the 1960's makes for a much richer database.  There are those however, who do not believe that including that data makes sense.  I disagree.

Having spent some time thinking about my pursuit here (creating a new passer rating system for the NFL), I've created new statistical measures in addition to those I created last year that I will discuss in upcoming posts.  I've also improved upon my original concept of CMI.

Anyhow, I'm glad to be back posting and look forward to a terrific 2010.

Cheers,
Kiran

First Year Quarterback Performance - An Evaluation Method

This past weekend, the New England Patriots traded Matt Cassel to the Kansas City Chiefs.  In fact, they traded linebacker Mike Vrabel and Cassel for a 2nd round pick (pick #34) in the upcoming April draft (see here for the details).  Depending on what your news source is, this was labeled as either a good deal for Kansas City, or a bad deal for New England.  The experts discussing this issue don't seem to support their arguments by any objective measures - they just opine.  I am not going to jump into the debate here.  I will offer this however - we will probably not know who will end up on the better end of this deal until many years from now, and even then, we may not know.  

What about the objective evidence?

Matt Cassel is not the only quarterback in 2008 that had a good year as a first-year quarterback.  There was Aaron Rodgers of the Packers, Shaun Hill of the 49'ers, and Matt Ryan of the Falcons.  There were four other quarterbacks who threw enough passes this year to make the list of first-year quarterbacks - Ryan Fitzpatrick of the Bengals, Joe Flacco of the Ravens, JaMarcus Russell of the Raiders and Tyler Thigpen of the Chiefs.  

Notice here that I didn't use the term rookie quarterback, but instead used the term first year.  The way I look at a quarterback, is that I only consider a quarterback who has thrown "enough" passes to qualify in a given year.  The # of passes has varied by year (as opposed to the fixed standard used by NFL.com).  For the 69-year period from 1940-2008, my database contains 1,451 passing seasons, 336 of which were by a quarterback qualifying for the first time.

How did the aforementioned 8 quarterbacks do?  

I use a statistical measure I have previously developed and discussed, called CMI (Completions Minus Interceptions, calculated as Completions/Attempts - 3*Interceptions/Attempts) instead of the NFL Passer Rating system (using the NFL passer rating system gets you essentially the same answers), and then use the standard deviation from the mean as the measuring stick (for the pool of qualifying passers, I calculate the the mean and standard deviations for each year, and relate a particular performance in that year to the mean using that year's standard deviation).  Using the mean and standard deviation this way not only allows me to compare these 8 quarterbacks relative to each other, but also relative to all quarterbacks this year, or any other year.

In a post not too long ago, I showed how, over the past 69 years from 1940-2008, when the standard deviation is used as the measuring stick, the accumulated data form a near-perfect theoretical standard normal distribution.  And I also showed how, this is true regardless of whether one used the NFL passer rating system or CMI as the statistical base.

I can now show you how these first year quarterbacks did compare to the rest of the QBs in 2008, using this measure.  But this picture may not tell us much, if anything.  Here's the graphical illustration:

Perhaps the only thing that may be apparent from this is that first year quarterbacks don't do nearly as well as quarterbacks with more experience.  Even drawing that conclusion from this is a little iffy.  However, it is clear (at least by this measure) that Cassel had the best year of the 8 first year QBs.  Rodgers was a close second.  JaMarcus Russell and Tyler Thigpen had disastrous first years.  In an earlier post regarding Jeff Garcia, I had suggested that Kansas City might be a good place for Garcia.  Scott Pioli, the new Chiefs' GM apparently was thinking the same thing - upgrade at quarterback, and he did - with Cassel.

Going back to the standard normal curve, let's look at just the first year performances.  Recall that, when all 1,451 seasons are aggregated in a histogram, it looks very much like a standard normal curve.

This doesn't quite look "normal", does it?  The first year performances appear skewed to the left, or, compared to all quarterback performances, appear to under-perform.  Since the total database is normally distributed, then performances in years 2 and beyond must obviously be skewed to the right.  This prompts the question: does experience improve performance, and, in particular, how and when?

One way to look at this is to take the entire database, and look at the average standard deviation from the mean for each additional year a player qualified.  Here's what that graph looks like.

Wow!  I didn't realize that it takes about 5 qualifying years to turn into an average quarterback.  And that's just for those playing that long!  Even though this graph goes out to 18 years, there is hardly any data beyond the 15th year (there are only 8 quarterbacks who have had qualifying seasons 16 different years).  According to this illustration, basically, first year quarterbacks, as a group, struggle mightily.  They then gradually improve each year until about year nine.  Quarterbacks, on average, must still endure sub-par years in years 2-4, and for those that survive, their reward is about four more years of improving performance, assuming they stay healthy for that long.

Note: This analysis - the method of breaking down a quarterback's expected career in terms of passing performance as it relates to experience and how it is expected to change over time, is the first I've seen published publicly.  But I digress.

Ok, so we've taken a look at these 8 QBs relative to all quarterbacks in 2008, and we've taken a look at first year quarterbacks in general.  So the next step is to take a look at the best first year seasons.  In my database, there are 336 first year seasons in all (in other words, in the 69-year period from 1940-2008, 336 different quarterbacks threw enough passes in a season to qualify at least once).  The table below shows the 75 best.

Cassel (#48) and Rodgers (#60) are the only 2 of the 8 in 2008 that crack the top 75 all-time.  On the other end of the scale, we get JaMarcus Russell (#264) and Tyler Thigpen (#290).

When Matt Ryan (#128) got off to a solid start, and the Falcons surprised many by getting into the playoffs, there were many debates as to whether his season was one of the best ever by a first year player.  Not according to this measure.  Often, during those discussions, Kurt Warner's first year was brought up.  In looking at the table above, well, he did have an outstanding first year in 1999.  It ranks #6 all-time in terms of first year seasons.  The best ever - Roger Staubach's 1971 season (Staubach was a rookie in 1966, and played in both 1969 and 1970.  However, he didn't attempt enough passes in either year to qualify).  

Notable quarterbacks high on this list: Joe Montana in 1980 (#4), Tom Brady in 2001 (#14) - (see my post earlier comparing Brady's season to Cassel's here), Brett Favre in 1992 (#15), Dan Marino in 1983 (#21), and Johnny Unitas in 1956 (#22).  

Notable absentees:  Steve Young in 1986 (#154), Peyton Manning in 1998 (#208), Joe Namath in 1970 (#202)*, and Troy Aikman in 1989 (#311).  The all-time worst first year performance - Terry Bradshaw in 1970 (#337)*.

* Keep in mind that 1970 was the first year of the NFL following the merger, and 15 quarterbacks showed up in the database as having that year as their first year, although Bradshaw's 1970 season was his actual rookie season.

Are we done?  Hardly.  So after I went through this list, and still trying to objectively value Matt Cassel, I was interested in answering the following question(s):  what does the first-year tell us about a quarterback's future potential?  Do quarterbacks who have good first years, have good careers?  What about quarterbacks who have sub-par first years?  In other words, is there a correlation between year 1 performance and subsequent years?  When in doubt, it's always a good idea to take a look at the data.

Whoa!  The correlation between year 1 performance, and career performance, as measured by CMI standard deviations relative to the mean, is 0.75.  What that really means is that 57% of the variation in a quarterback's career performance can be explained by his year 1 performance.  I find that incredible.  So I looked a little deeper.  It turns out that 112 of the 336 quarterbacks only have 1 qualifying year.  And, for that group, the correlation coefficient is, well, 100%!  If I only looked at those quarterbacks whose career included at least 2 different years in which they threw enough passes to qualify, then the correlation coefficient drops to 64%.  Still quite remarkable.  

The table below breaks down the 336 first year seasons in terms of CMI standard deviations to see if we can glean any additional insight into whether a good first year translates into a good career and vice versa.


Well, the evidence is pretty clear.  Taken as two groups, players that have better than average (remember, the mean and standard deviation is relative to all quarterbacks, not just first year quarterbacks) first years tend to have longer careers and more productive careers than those that have sub-par first years.  And, breaking it down even further (in other words, just looking at the group whose first year was better than average, or the group whose first year was worse than average), you can clearly see that even within those sub-groups, the better the first year, the longer the career (although this correlation isn't nearly as strong), and the more productive the career. 

I haven't spent too much time on my blog discussing individual careers, except in the posts discussing Brett Favre and Jeff Garcia.  If I do this en masse, it would be such a powerful post - I'll basically be giving you my list of greatest passers to ever play the game, that I need to put a lot more thought into it than I have to-date.  In any case, some of the obvious names will show up on that list.  However, you'll find some exceptions as well (and I'm sure many of you will take exceptions to the list).  The point I really want to make here is that, when combining the table above, and the discussion following the list of the top 75 first years, it is quite remarkable how Troy Aikman "escaped" the trend, and turned into a great quarterback (or, more accurately, a great passer) - he truly turned out to be an exception (as for Young and Manning, even though they didn't rank well on the first year list, their first years weren't "that bad" compared to other first year seasons - as a matter of fact, Young's was "above average" compared to a typical first year).  Notice how I don't mention Bradshaw and Namath - it is because, statistically, their careers were sub-par.  When I publish my list of all-time greatest (and worst) passers, we'll get into it in more detail.

So, in trying to look at Matt Cassel's first year performance objectively, we've looked at how quarterbacks perform as their careers progress, how quarterbacks perform in their first year, who had great first years, where Cassel's season ranks on that list, and, how a first year translates into (or, is predictive of) a career.  Now perhaps, we can view Matt Cassel's 2008 year in an objective manner, and, if so inclined, you can at least evaluate the trade armed with some data.

Naturally, while I was putting this post together a question popped into my mind, and that is:  is a quarterback's second year a good indicator of their career?  What about looking at their first and second years?  Check back in a week or two to find out.

Is Peyton Manning Having a Good Year?

Peyton Manning had a great game last night against the Jaguars, leading the Colts to a 31-24 victory, and a spot in the playoffs.  That makes it three games in a row now that he hasn't thrown an interception.  During that span, he has completed over 80% of his passes.

However, it seems to me that he hasn't received his proper due this year.  Perhaps it is because he has been rather mediocre until this recent stretch.  He did have three terrible games (week 3 at home against these same Jaguars, where he threw 2 picks and the Colts lost 23-21, week 7 on the road against the the Packers, where he threw two picks, and the Colts lost 34-14, and week 13, on the road against the Browns, where he again threw 2 picks, yet the Colts won 10-6).  He has thrown 2 picks in two other games this season, making it a total of 5 games this year where he has thrown at least two picks.  To truly appreciate how good Manning has been, consider this: Since 2002, he has never had a season where he threw 2 or more picks in a game in more than 2 games.

The table below compares each of Manning's years in the NFL.  For sake of comparison, I have included the league average CMI for qualified passers only - i.e. those who attempted at least 256 passes in a season (224 for the 14 games completed so far in the 2008 season).  The chart illustrates that the league average CMI has been steadily increasing during these years, and Manning continues to perform better than the league (he has done this every year, except for his rookie season - 1998).

From this data, I would conclude that he is indeed having a good year, at least as compared to the league.  He is having a typical year for him.

In the table below, the column "Manning CMI rel to League Avg" is calculated as follows:

Manning CMI Rel = [Manning CMI - League Avg CMI] / Std Dev of League Avg CMI

For example, for 2008,

Manning CPI Rel = [0.599 - 0.542] / 0.052 = 1.08

CMI - 1940 to 2007

I thought I had posted this earlier.  It is a set of graphical illustrations of CMI from 1940 to 2007.  For comparison sake, I have included NFL average QB Rating in the 2nd graph.  CMI is defined as follows:

CMI = Completions/Attempts - 3 * Interceptions/Attempts

This ratio can never exceed 100.0% (although it can be negative).

In my mind, a running back has 2 duties once he receives the ball - run with it, and don't fumble it.  Similarly, a wide receiver has 2 duties once he receives (catches) the ball - run with it, and don't fumble it.  And, a quarterback, similarly has 2 duties - complete the pass, and don't turn it over.  I realize it's too simplistic a view.  Eventually, I will incorporate sacks and fumbles into the formula.  Since I am very interested in comparing quarterbacks over time, once sacks are included, I can only go back in time to 1969 (since sack data was included).

Best 25 Seasons in History - CMI

Here's a list of the 25 best seasons in history in terms of CMI.  This list is different than a similar list I published earlier, based on QB Rating.  As I mentioned earlier, when looking at career CMI (versus career QB Rating), this list favors those quarterbacks who didn't throw a lot of interceptions.  What is particularly interesting, is Tom Brady's 2007 season.  He is rewarded on the QB Rating list (#2) for what he did - he threw 50 touchdowns.  He is rewarded here for what he didn't do - he didn't throw a lot of interceptions.  Also, interesting is that, according to this ranking, Peyton Manning had a better 2003 season than he did in 2004 (although he gets a lot of attention for what he did in 2004).  Again, this list is biased towards the more recent years, which of course suggests Sammy Baugh's season in 1945 must have been out of this world.  I will later on, look at these figures, relative to the time periods in which each of these seasons took place. 

Highest Career CMI

The two tables below show the Top 100 quarterbacks in terms of their career CMI. Surprising? Well, it should be. It doesn't meet with your expectations. Chad Pennington the best quarterback in history, terms of CMI. Well yes. And no. Keep in mind that Chad Pennington is no slouch - he has the 8th best all-time career QB Rating. As a matter of fact, the quarterbacks who have top 10 highest QB Ratings in history, are in the Top 10 in terms of CMI.


I have shown before, that CMI has been steadily increasing over time. So obviously, QBs in more recent years have had higher CMIs than earlier quarterbacks. But the list below does give us a different picture of the top quarterbacks in history compared to a similar list ranked by QB Rating.


CMI does not reward quarterbacks that throw a lot of touchdowns per se (see Dan Marino). It does reward quarterbacks that do not throw a lot of interceptions (see Troy Aikman). Think about the teams that these two quarterbacks played on. Dan Marino never had a quality running back for an extended period of time. Troy Aikman had, well, Emmitt Smith, he of the NFL-record 164 career rushing touchdowns. CMI doesn't penalize a quarterback for sharing the backfield with a good running back.


My goal is to eventually identify the best passers in history, using CMI. The way to do it (in order to account for the ever increasing average CMI), is to relate each quarterback to his peers. I am currently working on methodology that can do this "properly".

Crazy, Debatable and Lucky

Today, the Green Bay Packers lost at home to the Houston Texans, 24-21 on a last-second field goal.  This dropped the Packers to 5-8 for the season, and, for all practical purposes, knocked them out of the playoffs.  I'm sure the Packer faithful will start to think what might have been....

The Packers were coming off a 13-3 season, having won their division handily, and were one win away from the Super Bowl, a loss to the eventual winner, the New York Giants at home.  

Then the soap opera started.  First, Brett Favre, the Packers QB for the past 16 seasons, decided it was time to retire.  The Packers, who had drafted Aaron Rodgers in 2005, were prepared.  During the offseason, the Packers named Aaron Rodgers the starting QB for the 2008 season.  Second, Brett Favre decided he wanted to play in 2008.  Nothing wrong with that.  Only that he had "lost" his starting job to Aaron Rodgers.  No way a future hall of famer is going to be carrying a clipboard for an untested, essentially rookie QB.  Not after the season Favre just had had.  After much ballyhoo, the Packers traded Favre to the New York Jets.  In return, the Packers received a conditional draft pick.  For now, it's a fairly safe bet that it will be at least a second rounder.  The Jets, who finished a lousy 4-12 season, now having found their "saviour", then released Chad Pennington.  Of course, the woeful Miami Dolphins, having just completed a miserable 1-15 season, and desperately needing some talent, promptly picked him up.

So there we have it.  The Packers make a "crazy" move, going with untested Aaron Rodgers, over Hall of Famer Favre.  The Jets, in need of a new direction, trade for Favre and release Pennington in a debatable move.  And the Dolphins, well, got lucky.  Pennington is a dramatic improvement over any Dolphin QB.

Ok, so that's how we all thought about it back in the beginning of the season.  So, let's take a look at how the decisions panned out, in terms of their actual performances.

The table at the top of this article shows each player's stats for the 2008 season, the 2007 season, and their career #s.

First, let's start with Rodgers.

He is having a very good year, despite the team's troubles.  His QB Rating is a robust 92.1 (league average is 85.3), and his CMI is 0.560, about 0.4 standard deviations above the mean of 0.538.  Not bad for being a first-year starter.  Compare him to three other first-year starters here.

Now, let's take a look at Favre.

He had a career year in 2007.  His QB Rating of 95.7 was only exceeded in his MVP years of 1995 (99.5) and 1996 (95.8).  His third MVP year, 1997, his QB rating was 92.6.  Favre's career QB Rating (leading up to the 2007 season) was 85.0.  It has since improved to 85.9 through 13 games of the 2008 season.  In terms of his CMI, 2007 was by far and away, his best year (0.581), eclipsing his previous best of 0.561 in 1995.  Given that it had been a long time since he had put up that kind of #s, it would be reasonable to assume that it wouldn't necessarily continue into 2008.  Well, he's having a (surprisingly) outstanding year in 2008.  His QB Rating is a healthy 88.2, and his CMI is 0.577 (almost as good as last year).  Is he why the Jets have improved?  Hard to say.  Especially, when you take a look at what Pennington has done this year.

Pennington's #s:

Chad is easily having his best year since his first full year in the league, in 1992.  Back then, he produced a QB Rating of 104.2 and a CMI of 0.644.  Only Tom Brady in 2007 (0.647), and Hall of Famer Troy Aikman, back in 2003 (0.645), have had better seasons.  This year, Chad's #s are a QB Rating of 93.7, and a CMI of 0.618.  Compare these to his career #s of 89.7 and 0.578.  By the way, Pennington's career CMI of 0.578 is the all-time career NFL mark.  Next is Steve Young at 0.565.  Pennington's #s are also, needless to say, a dramatic improvement compared to those of the 2007 Dolphins (QB Rating of 67.4, and CMI of 0.484).

So, what does this all mean?  It means that the Packers weren't crazy when they decided to put their faith in Rodgers, the Jets decision is debatable in giving up a 2nd round pick in 2009, and exchanging a QB who, while having an outstanding year, is not doing as well as the QB they gave up on, and the Dolphins got awfully lucky to end up with Pennington.

A Slight (but necessary) Digression

Brian Burke, who does some excellent analytical work at advancednflstats.com has made this observation regarding three first-year quarterbacks - Joe Flacco of Baltimore, Matt Ryan of Atlanta, and Matt Cassel of New England.  He suggests that their performance has been improving as the 2008 season has moved on. and shows their performance, graphically illustrating it in the following manner.  He uses a measure, "Adjusted YPA" or, Adjusted Yards per Attempt, defined as [Yards -40*Interceptions + 10*Touchdowns]/Attempt, and then looks at their 4-week moving average to compare the three quarterbacks.

Well, I decided to look myself.  I have been "casually observing" that Flacco and Ryan have been having some "pretty decent" games recently (Ryan is on my Fantasy team).  So, I decided to put my newly developed CMI to the test.

Here's what it shows:

It appears to be very consistent with what Brian found.  Matt Ryan has improved the most.  CMI shows that Matt Cassel has in fact had the least improvement, if any.  However, to his credit, he has been performing at a very high level from the beginning.  Looking at the table below, you see that both Ryan and Flacco have remarkably similar attempts and completions through the first 12 weeks, with their only difference being the # of interceptions thrown.  Also, you can see that Cassel attempts more than 5 passes per game more than the other two (partially reflecting the fact that New England has no running game).

Brian has, in the past, ventured into the "creating a new passer rating" space, and, in fact, created one.  He uses a concept called "Air Yards", defined as, [Yards - Yards After Catch] to relate "passer rating" to a team's wins and losses.  See his post here.  I agree with his approach generally.  In other words, if you look at his formula, it's a lot like CMI.

QB Wins Added = [(Air Yards - Sack Yards) * 1.56 - INTs * 50.5]/Pass Attempts - 3

where, (Air Yards - Sack Yards) is substituted for Completions.  In other words, he also takes out Touchdowns thrown and Yards per Attempt.  Brian is the first person I know of that does not include touchdowns as a part of a passer rating calculation, and he is to be commended for that.  My only real issue with Brian's calculation is that YAC, or, Yards After Catch, is not a readily available statistic, and hence, "Air Yards" is not easily calculated.

I am still developing CMI, and the next (enhanced) version will incorporate sacks.  Given that one of my goals is also to look back in time and be able to compare QBs over time, I will only be able to that back to 1969.

As I mentioned at the top, Brian is a very smart guy, and has done some very good work on a whole host of topics relating to the NFL, with particular attention to statistical rigor.  I will be referring to his site quite a bit in future postings.