Chris Tomasson, reporter for the Pioneer Press, loves to bring up Pro Football Focus grades for better or worse when talking to and about players and he did so once again when talking to struggling Vikings offensive tackle Matt Kalil (ranked as their third-worst offensive tackle). Kalil, like head coach Mike Zimmer had earlier in the season, didn’t take too kindly to Pro Football Focus’ grades (or their use in reporting, blogging or generic fan impressions).
“You can’t listen to some people who don’t know what they’re talking about, who want to go on Pro Football Focus, the website who they hire random fans and give them a training camp on how to grade people,’’ Kalil said Thursday. “And people want to live on that site when it’s not really a credible site.
“I’m not going against them because they hammer me. But I mean, it’s not a credible site. … They don’t know the blocking schemes, they don’t know who’s fault (a bad play might be).’’
I have no issue with Tomasson’s use of PFF; it supplements a lot of my blogging, and is a fantastic resource. Like any resource, it can be a little cumbersome if used uncritically or without context, but it’s a bigger asset than a detriment even if it has its own limitations. While I think Tomasson is particularly subject to the limitations in the way that he uses it, it’s not as if he’s all the worse a reporter for PFF’s existence.
I think the only risk with PFF usage is not the kind of risk that people engage in all too often—using those grades as gospel can be an issue for those doing serious analysis, but it’s also not something I see happen on a consistent basis. Besides, if it’s good enough for Brian Robison, shouldn’t it be good enough for Matt Kalil?
I enjoy reading the PFF articles because they go into more detail than just the numbers. In our day and age numbers get you the accolades, but don’t always tell the whole story. – Brian Robison, in PFF testimonials
Sam Monson of Pro Football Focus defended their website and practices in the piece.
“Individual players are, of course, entitled to their opinion but we are a little more credible than a group of fans guessing every Sunday,’’ Monson wrote in an email. “We have sat down with coaches, front-office personnel and players and explained to them what we do. We have graded plays with them and they’re on board with the system, the accuracy and what we do. … Our recruitment and training is also a little more sophisticated than an analysis boot camp.’’
Kalil in the piece talks about his father (a former offensive lineman) and brother (a current offensive lineman for the Carolina Panthers) and how they provide critical feedback. At the same time, it’s a little difficult to take everything Kalil says about his feedback without a grain of salt. For example, Ryan Kalil mentioned that Matt is “his own harshest critic” in an interview before the Vikings-Panthers game.
Ryan Kalil said he talks to Matt after every game. “He’s his harshest critic.” Talked about LT being tough when #Vikings get behind.
— VikingUpdate.com (@VikingUpdate) November 26, 2014
Matt Kalil seems to have been particularly sensitive to the criticisms he’s heard all year, so this is a bit difficult to believe—especially because Matt goes out of his way in the Pioneer Press piece to say that things are “blown out of proportion.”
Further, the idea that football is a “black box” that is difficult for outsiders to divine is one of the most obnoxious and untrue myths in football, and it lends artificial credibility, and conveniently a shield from scrutiny, for coaches on the hot seat or players who are underperforming at positions that do not have easy statistics or highlights to look up.
This ignores the fact that there are former offensive linemen (and other football players) across the country who are perfectly willing to become analysts and criticize players along the same lines that amateurs like myself or theoretically those at Pro Football Focus do. Football is often an open book, and even though there are a number of things that are esoteric to the highest level, they are often oriented around specific techniques and not evaluating outcomes.
For example, I don’t know what kind of footwork Norv Turner prefers in his quarterback dropbacks, but I do know that it is typically deep and often aligns with the seven-step drops that he has become somewhat famous for. There are a lot of different types of footwork designed to alter timing, depth and throwing mechanics on different reads (say, quick-five versus big-five) and differences in the amount of time that a quarterback will be asked to hold on before the throw (how many hitch steps, for example, should a quarterback have on a particular play if the protection holds up?) that all inform technique and are relatively exclusive to the quarterback, offensive lineman, safety or whatever that are not easy for me or many other people to comment on.
But that doesn’t mean evaluating the outcome of a play is particularly difficult. For example, I do not know why Matt Kalil’s play on this snap was so bad, but I do know that it was:
Was his footwork wrong? Was he slow to move upfield by design or by limitation? Was he taking extra care to get rid of the inside move? I don’t know. What he did was wrong, and it is fairly obviously his responsibility. Whether or not his technique on that play was not enough to stop a wide rush (where offensive linemen are consistently taught to drop deep regardless) or his guess on Jared Allen’s counter-move was wrong, he didn’t do his job, where in previous years, he did.
This sort of thing pops up constantly for players like Matt Kalil. Kalil has been penalized 11 times this season, the second-most of any offensive tackle. Did the Vikings call the “Kalil should false start” play? I’m not trying to be snide, so much as attempting to demonstrate that there are very few instances where an informed analyst grades a play poorly because they misunderstood what happened. Matt Kalil has allowed sacks to players he’s clearly actively trying to stop, and allowed sacks to players without other players (offensive or defensive) in the same area code, which would force us to ask what exactly his assignment was.
It’s not as if PFF is flying by the seat of their pants, either. I wrote about the use of PFF in player evaluation in the offseason when I argued that Chad Greenway was not a particularly good linebacker.
Every time someone mentions Pro Football Focus, another person has to mention that PFF are just “box score watchers” or that they are obsessed with stats. Not only is that the opposite of true, but agents regularly use PFF in their player negotiations (and of course, there are some front offices that are not agreeable to this, but many that are). NFL teams purchase and use PFF’s player tracking data, though do have their own internal grading systems.
But back to the point: PFF watches plays and grades them. It doesn’t figure out if a player earned a stat when it grades them, it sees if a player made a good play or a bad play and then gives them a score from -2.0 to +2.0 and adds them up at the end of the game.
Those grades are not statistics, they are subjective evaluations. But not all subjective evaluations are made equal: some are done with more information and rigor than others. PFF is the opposite of a highlight reel, and regularly informs itself through conversations with NFL teams about responsibilities and schemes.
It’s difficult to emphasize this enough—Pro Football Focus is not a random collection of people with too much time on their hands. They make most of their money not by selling individual subscriptions to people like you or me, but providing a wealth of data to NFL teams—more than we have access to with their premium subscriptions.
In return, Pro Football Focus will receive feedback from players, coaches and front office personnel about the way that they grade and what evaluators are looking for. Here is an example of a former actual NFL agent in discussions with a former actual front office personnel man conducting a mock negotiation over J.J. Watt’s then-hypothetical contract—a mock negotiation designed to explain to fans what actually goes on in meeting rooms:
Conventional statistics don’t do J.J. justice. Football Outsiders has a metric called “Defeats” that measure turnovers (or tipped passes leading to turnovers), tackles for loss, and tackles or passes defensed on third or fourth down. J.J’s 56 Defeats in 2012 were the most since Football Outsiders started the metric in 1996. According to Pro Football Focus (PFF), J.J. batted an unheard of 15 passes in 2012. Nobody else in the last five years has reached double figures in batted passes.
In some respects, J.J.’s 2013 campaign was better than his 2012 season when he was the near unanimous NFL Defensive Player of the Year. He had 85 total quarterback pressures (combined sacks, quarterback hurries and quarterback hits) in 2013 according to PFF after posting 76 pressures in 2012. J.J. also graded out better in 2013 than 2012 with PFF. His 111.6 grade was 10 points higher than his 2012 grade of 101.6. The next highest grade for a defensive player in the last two years is Geno Atkins in 2012 at 80. J.J. is the modern day equivalent to Reggie White, but without a contemporary who is a modern day Bruce Smith.
. . .
Since there aren’t 10 non-quarterbacks better than J.J., these figures are just more evidence of how inappropriate the deals you have been proposing are for a player of J.J.’s caliber. For your information, the average yearly salary of the top five non-quarterbacks using the parameters is $15,457,667. As you may not know, J.J. heads Pro Football Focus’ list of the NFL’s top 101 players for the second straight year. J.J.’s ranking is further confirmation that he doesn’t have any peers. The average of the top five non-quarterbacks also isn’t an appropriate gauge for J.J.
This isn’t an isolated instance. PFF has on their website a list of testimonials from former players, current agents and heavy hitters in the industry. There’s a reason former All-Pro receiver Cris Collinsworth purchased a stake in Pro Football Focus:
This is an interesting story. From his home in England, Neil Hornsby, one of the biggest football fans on the planet, and his staff at Pro Football Focus have made a business of analyzing the performance of every player on every snap and grading their performances against every other player at the same position. Born in 2006, Pro Football Focus has done such a thorough job that 13 teams have contracts with the company to provide the kind of data on players and formations they can’t get anywhere else.
So this spring, Collinsworth was looking to improve his data collection and signed up for an annual insider’s subscription to PFF. The free part of the site looked real, but he wanted to know more, and so he filled in the email contact form asking for someone to get back to him. Hornsby, with his thick English accent, phoned up. “I thought I’d been scammed,’’ Collinsworth said. “So I figured I would ask him a bunch of questions, you know, to expose him … and after about five minutes I could tell he was absolutely legit, so I just shut up and listened to Neil talk about football at a very high level.”
At first Collinsworth said he wanted to simply get to know more about all 32 teams without having to do the kind of film study and painstaking research that would take a couple of days at least—in his words, “to be in position Monday morning to know what would normally take me till Tuesday night to know.’’ Collinsworth said he was actually thinking of engaging some people to watch NFL tape and grade it for him until he realized PFF already did it, and did it accurately. So last week he reached an agreement with Horsnby to buy a significant stake in the company. Hornsby and his crew of 12 full-time graders continue, and Collinsworth will use his influence to help grow the company and seek out new business for it.
There was one final test for Collinsworth. He and Hornsby separately graded a game. When they compared notes, their grades were very close. “And when we went back over them,’’ Collinsworth said, “I’d say on 50 percent of the discrepancies, they were right and I was wrong.”
The deal came together in a couple of months this summer, culminating last week. “What really impressed me,’’ said Collinsworth, “is the fact that 13 NFL teams have contracted with Pro Football Focus for their data. I mean, I have been around the NFL for over 30 years, I know how hard it is to get behind the wall of those teams. And they’ve got 13 teams to trust their data. That’s huge.”
“What appealed to me was Cris is interested in football for the right reasons—for pure football reasons,” said Hornsby, who has been a guest on my training camp tour for the past several years. “People like Cris want legitimate content. And I think he’ll help us get in front of other people who realize that legitimate content has an important place with the public today.”
I understand that you have to take those testimonials for what they are: advertisements. Collinsworth saw a business opportunity, took it and talked it up. Maybe he’s misrepresenting, but I doubt it. Even when you scrub the testimonials from their website and the fact that Collinsworth purchased a stake in a company and then promoted that company, the conversations I’ve had with football people are real, and when they’re not being criticized by Pro Football Focus, they are fans.
I’ve talked to several Vikings players (many not on the team anymore) what their best and worst games were. It just so happened, when I looked up those games on Pro Football Focus, their recollection matched Pro Football Focus’ grading nearly to a T.
But of course when I asked Charlie Johnson in training camp about Pro Football Focus negative grade of his play in the last two seasons, he was dismissive almost for the exact same reasons Matt Kalil was.
“You know what, I don’t know,” he said when asked why there was a difference between the grades he’s received and his evaluation of play. “it’s something that you deal with. I think if you go across the league, there’s always guys who get talked about. You pay no attention to it. The only thing I care about it is the way my offensive line coach, the way the head coach, the way the coordinators and front office think and if they’re happy with me, then I’m happy.”
Charlie Johnson went on to say that people at independent grading agencies “aren’t in the meeting rooms” and “don’t know the assignments,” when it comes to evaluating his play, and that the Vikings are very happy with his play (they did bring him back, after all).
If both Kalil and Johnson are playing better than PFF gives them credit for, who the hell is giving up all the pressure from the left side? Are they being asked not to block players? Someone gave up 17 sacks, 12 hits and 54 hurries from the left side, split up between the two in some way. Even if you split them evenly—8.5 sacks, six hits and 27 hurries—Kalil would have given up the most sacks of any left tackle in the NFL, and 52nd of 57 tackles in Pro Football Focus’ pass blocking efficency grades.
What’s worse is that Charlie Johnson would then look even worse than he does now, and he ranks 57 of 58 in the same statistic among guards. This time, he would be the league’s worst in pass protection by a mile.
It’s someone’s fault, and on almost all of those sacks, there wasn’t a tight end on that side of the field.
Whether or not Matt Kalil or the Vikings like it, PFF is part of the national football conversation, and that conversation has been enriched because of it. I’ve talked to multiple NFL agents (and you can ask some of them on Twitter yourself) about the use of PFF in negotiations. It turns out that almost all of them use them often, with various receptivity from front offices for that point.
Not only is PFF fairly accurate about their evaluations, they are relatively open about them. One thing that may be levied as a legitimate criticism of PFF is that they aren’t completely transparent about how they make grade changes after the release of All-22 film on Wednesdays, but they do change grades in response to new information until they have all available data. PFF’s score is usually very accurate on Monday morning, but is often completely on-point by Wednesday night.
That said, if you email them, you can get why they graded a player positively, negatively or not at all.
At the end of the day, it doesn’t really matter. Whether or not Pro Football Focus is completely accurate in their game-to-game grades or not, it is exceedingly unlikely that they would rank a player as one of the worst at their position when they in actuality were average or even simply below average. He has been performing poorly, a fact he occasionally acknowledges but always backstops with “but it’s all blown out of proportion.”
Even if PFF was completely wrong, the coaching staff has openly acknowledged his struggles. He could put the issue at rest (and take responsibility for it at the same time) by simply saying he knows he’s struggled and needs to perform better, even if the grades were a crock. Other tackles have done the same. The best example is Bobby Massie, who was Pro Football Focus’ second-worst tackle of 2012 through eight weeks. His PFF grades, to him, were a stark realization that he was performing poorly.
Bobby Massie opened his computer in early October and the numbers were staring back at him.
Only one other tackle in the NFL was ranked lower than him, and it happened to be a teammate. The number of sacks Massie allowed was highest in the league.
It hit home. It was time for Massie to change.
It all happened the week leading up to the Cardinals’ trip to Green Bay. Massie, the rookie right tackle, had, by all statistical accounts until that point, struggled this season. It was one thing for the coaches to ride him or the hometown fans to get discouraged but it was different for the entire league to see the numbers posted on ProFootballFocus.com.
Massie turned it around in a big way:
He didn’t allow a single sack in the second half of the year. He didn’t end up winning the camp competition in 2013, but he did see 57 snaps on the field without allowing a sack (only two hurries) and ended up the starter for the Cardinals this year, ranked as an average right tackle. Perhaps not a huge comfort to a team that has as of this writing lost two of their quarterbacks this season to Rams defensive ends, but definitely an upgrade over where Massie was.
If it sounds like I’m emptying the tank just to defend Pro Football Focus, it’s because it’s asinine the kind of stuff that an extremely accurate, useful and thorough organization goes through because they’ve done more to illuminate the game than almost any organization out there. They don’t need defenders, but I know how much work they do and it is massive. To dismiss the immense amount of knowledge, expertise and self-reflection the group of graders at Pro Football Focus does on the altar of the myth of “the mystery of football” is insulting.
Football is fascinating and endlessly complex, and fans do not know whether or not the protection was half-slide with big-five-and-a-hitch drop depth or if the switch concept on the open side of the formation was supposed to have a receiver check into a hitch route against a zero cover blitz. But they do know if an offensive tackle clearly attempts to block an outside linebacker and fails, he did not do his job.
Matt Kalil already looks like a sensitive, out of touch asshole after knocking the hat off a heckling fan after the Packers game (which he attributed to losing Phil Loadholt and not frustration from his penalty-ridden play). The fan certainly was classless, but that doesn’t really excuse Kalil’s behavior at a time it seems like he won’t take responsibility for his play on the field.
He didn’t bring up the PFF issue himself. But he could have resolved it by simply acknowledging he was playing poorly. If this is part of a bit where Mike Zimmer and the other coaches are trying to keep his confidence up and therefore encourage him to play better, it’s not working (or rather, it’s not working enough—the last eight games have been better than the first six, but they are still bad). What was at stake in the question was what steps he was taking to improve his play, not the credibility of a third-party organization. In this case, it may be better to acknowledge reality instead of nitpicking those who are pointing it out.
Or maybe the Vikings could risk their new quarterback. Whichever works for Matt.