Editor’s note: How efficient is your favorite — or least favorite — quarterback, really? There’s a new way of looking at the position, now that ESPN Stats & Information Group has created The Total Quarterback Rating (Total QBR). The stat measures all of the QB’s significant contributions and how they affect his team’s performance.
The new rating will be formally introduced during an hour-long ESPN Year of the Quarterback SportsCenter Special: The Total Quarterback Rating on Friday (8 p.m. ET, ESPN).
The measurement will be featured throughout ESPN’s comprehensive NFL coverage on Monday Night Football, NFL studio shows, and across other multimedia platforms.
Front Row interviewed Jeff Bennett, Senior Director, Production Analytics & Research, for more insight into Total QBR.
FR: How did the idea come about for QBR?
JB: ESPN created the Production Analytics team in fall 2010 to not just keep up with the niche websites that do advanced stats, and the front offices and third-parties, but to leverage our data and our experts at ESPN and own some stats and develop them here. We scouted the world for some of the best talent. Two members of our team were already ESPN employees and another person we hired is Dean Oliver, who was director of quantitative analysis with the Denver Nuggets, one of the pioneers of advanced statistics for basketball who was also very plugged in to the advanced stats community across all sports. Once we brought our team together and looked at the opportunities we had as far as creating new metrics. Football was an obvious place to start. We have great data and being ESPN’s Year of the Quarterback, it just seemed like a natural fit to focus on the quarterback position.
The NFL Passer Rating is just that. It’s a system that looks at passers in the NFL. It’s been much maligned for many years but no one went up there to challenge it, to really take the time to do it right to create a better statistic that captures what a quarterback’s job really is, as far as directing the offense and trying to win games. We knew we had the opportunity here with our people, our data and our stable of NFL experts. Our goal was to leverage our analysts’ knowledge to create a stat that could be used to help ESPN tell the story of the quarterback position across all of its platforms.
FR: How do you explain Total QBR to the everyday fan?
JB: It’s a rating that looks at all facets of quarterback play. It accounts for the game situation of every play. It’s what they do with their arm and their feet. It’s when they do it and how that relates to their team’s chances of scoring and their team’s chances of winning on every play that they’re involved in. It even takes into account stuff that isn’t a play. Penalties aren’t official plays but quarterbacks who are drawing penalties or committing penalties, those are big events in games. So we’re counting those as things that are part of a quarterback’s job. When they draw the defense offside, there’s a skill to that. When they are penalized for delay of game, that’s a reflection of something and we’re capturing that as well.
FR: How were responsibilities broken up among the members of your group?
JB: Dean is really the brains behind the operation as far as the concepts and the theory to put this all together. He organized the data the right way. He set up the queries. Alok Pattani was our main number cruncher, essentially going through and creating the framework for this to run. While Dean was writing the algorithm and conceptualizing all the different pieces we wanted to capture, Alok was doing the mathematical analysis of all the down distance, using the real NFL play-by-plays and the results to create the tables necessary to power the rating. We also went out and hired a contractor named professor Ben Alamar from Menlo College in Silicon Valley. He is a consultant for the NBA’s Thunder and he’s worked with the 49ers on the relationship of the offensive line. He’s someone we knew from past conferences who could help give us a deeper understanding of the quarterback and his role. The fourth member of our team Albert Larcada was working on NBA and soccer analytics projects. The last six weeks or so, he’s really moved full-time to QBR to pull out all the storylines and really analyze and interpret the early results so we can start to figure out who the best quarterbacks are and why.
FR: How much is analytics used in football compared to other sports?
JB: Almost every baseball team has a numbers part of the organization. It’s well documented with teams like the A’s, Red Sox and others. With the NBA it’s becoming much more popular too. There are probably 17 teams that have some sort of quantitative unit amongst their front office. The Mavericks just won a title with a colleague of Dean’s named Roland Beech, who had a website 82games.com and got hired to do analysis for the Mavs. This year he was on the bench with the assistant coaches and Mark Cuban even gave him credit by name after they won the title. It’s also really growing in hockey. In football, it’s tight-lipped. The amount of teams working on this is really guess work. We know people like Ben who has worked with the 49ers. The Patriots are very progressive in this area and have been for a while. Some of the decisions they make from an outsiders perspective seem curious or controversial, but they are rooted in data. Bill Belichick’s 4th and 2 calls, the outcome was poor but it was actually better than a 50-50 proposition to go for it. The way they sign and get rid of people, they value players because of what the numbers say. They don’t put a lot of emphasis on running backs where other organizations do. They look at other positions of need. Our numbers bear that out too. The differences between a good quarterback and a poor one is a mile large and how much they contribute to team scoring. The best and worst running backs are much more tightly packed and they contribute on average far less to a team’s offense. There are teams that have caught on to that, and that gives them a competitive advantage.
Jeff Bennett and his staff have a new formula for rating NFL QBs.
FR: Can QBR analytics principles be applied to other NFL positions?
JB: One of the things that makes QBR so exciting is the way we put the framework together to power the rating. It’s based on how many points the quarterback is actually responsible for, distilled out from his teammates on the offensive side of the ball. In the background are buckets where the linemen, running back or receivers are worth this many points. We haven’t spent our time extrapolating that into a rating because we focused on the quarterback, but those elements are all in place. It’s all predicated on the same system. It’s also going to allow us to take these same principles down to the college game and do the same thing there. The data isn’t as rich in the college game. It’s a little different study based on the variety of offenses and the way the game is played, but it’s a logical step to go there as well. We may also look at extending this to a historical rating. We don’t have as much data going back in the NFL as far as games that are video-tracked, but we have a lot of stuff that we’ve learned from these studies about sack rates and things of that nature to develop a ratings system that transcends different eras.
FR: How did you begin to create the QBR formula?
JB: We tried to learn as much about the quarterback position as we could. What is the job of a quarterback? It seems simple but we wanted to know what they were doing on every kind of play. When they are passing, what information are they looking at? Are they under duress? How much time do they have in the pocket? How many yards after the catch are available based on how far the team is from the end zone? We’re trying to parse all this out, and we’ve talked to ESPN experts like Trent Dilfer, Jon Gruden, Ron Jaworski and Tim Hasselbeck to get their pieces of this and to help us with maybe what the data is not showing. Once we found out what they were looking at, we would find the numbers that are useful to tell that story, and then we do the analysis over the 60,000 plays since the start of 2008 to see what was significant about each of those pieces of information. It was really months of talking to our experts and studying the data.
FR: What surprised you most through the whole process of creating QBR?
JB: There were a ton of surprises, which is fun. Big picture surprise is how hard it is to create a statistic. It is HARD to do it right. We shot for the moon here and wanted to account for everything. To do that in a sport like football where it’s so situationally dependent, and the players are so intertwined on the offense, took an enormous amount of study, and we really needed the best and brightest minds to think of these problems with an open frame of mind and do the number crunching behind this to validate it. Putting it all together and seeing the results, I just knew we were accounting for things the right way.
When QBR data starts spitting out that the best quarterback over the last three years is Peyton Manning, it makes sense. But, the NFL Passer Rating has him as the fifth highest-rated quarterback since 2008 – and that’s just looking at passing, which is Peyton’s strength. How can he be fifth? In our system, he’s number one and it’s actually not even that close. I found that to be surprising – how dominant he is compared to Tom Brady, Phillip Rivers and Drew Brees. The gap is large, and that’s because Peyton does everything great. He does more in terms of air yards, he makes harder plays, he takes fewer sacks than anybody, and he never fumbles the ball. He just doesn’t have negative plays. Our research also indicated he is the most consistent quarterback since 2008.
FR: How do you expect QBR to be used on ESPN?
JB: We will be able to provide a live in-game rating for Monday Night Football, which is an enormous accomplishment based on how complex the formula is and the type of data that powers it from live video-tracking analysis. We will have multiple people [from ESPN’s Stats & Analysis Team] track different parts of the game, and we will be entering information up to the second to input into the system that creates the rating each week. Across ESPN, fans will be able to use it a variety of ways. ESPN.com is very interested in it for weekly leaderboards. We will also have editorial that will help fans learn more about it. We’ll have lists going back to 2008 so fans can pull from it and understand things like why Matt Ryan is better than Joe Flacco. A big reason for that is because Flacco takes so many sacks, which hurts his rating. Another neat point from our research is that per 100 plays for last year’s Broncos, Kyle Orton added a little over four points to his team. Tim Tebow for that same 100 plays added over 11 points. We know there’s basically a seven point difference over 100 plays – or the equivalent of three games – that Tebow, based on his running and his lack of negative plays, was a much better quarterback in helping his team score situationally. It’s a small sample size but we think fans will enjoy that kind of information and apply it to fantasy football and other things. On the TV side, we have researchers for NFL Live, Sunday NFL Countdown, our Monday shows and SportsCenter, among others, who are going to be well-versed in the rating to develop content pieces to help tell the story of the quarterback position. ESPN The Magazine and Insider are also very excited about this and will write about it weekly with Peter Keating and maybe a few others to provide analysis and tell stories that fans can’t get anywhere else.
FR: What do you expect to be the legacy of this project?
JB: This is a tremendous opportunity for my group personally. We started a group to basically create a new stat and the content group creates an hour-long primetime program about it. I don’t know that that’s ever happened before. It’s everything we hoped for and yet it’s also daunting. We see this as a great punctuation for all the terrific projects Year of the Quarterback has done this year. They are excited that this will live on and it’s something we’ll come back to years from now and say, ‘that all started here.’ I’m really thankful for the partnership we’ve had with Content Development and others. I’ve been here for 17 years and for me personally there’s nothing else that comes close to this project.
