Super Bowl XLIX Thoughts and Why Judging a Call is More Complicated Than it Seems

  |  
Feb 3, 2015
Photo Courtesy Seahawks.com

I know this isn’t Vikings news, but it’s on my mind and it’s my blog so I’m talking about it. The final play from the Super Bowl—or rather, the final meaningful play aside from some smart chicanery by Belichick to draw an encroachment penalty—will probably go down in history as one of the worst calls in sports, up there with Red Right 88.

The fact that Red Right 88 was probably a much more defensible call than fans would have you believe is besides the point, because the call for a slant off of a receiver pick is too being unfairly maligned. End-of-game decisions are the easiest decisions to model in theory, but present with them a degree of complexity created by the clock and the down system that actually make end-of-game decisions the most difficult decisions of all to model in real-time.

There are a couple of different points in the process where the Seahawks were presented with a decision, and throughout those different decision points, the Seahawks made defensible calls.

As the Seahawks were approaching 2nd and 1 from the goal line having just run Marshawn for four yards, the Patriots put in their goal-line package, a 5-3-3 setup with Rob Ninkovich, Chris Jones, Sealver Siliga, Alan Branch and Vince Wilfork on the line and Dont’a Hightower, Jamie Collins and Akeem Ayers as linebackers, but playing close to the line to the line of scrimmage.

The three defensive backs were Darrelle Revis, Brandon Browner and Malcolm Butler (some nobody that worked at a Popeyes I guess).

There are some initial thoughts here, the first of which is that Bill Belichick should have called a timeout to preserve the time in the event that the Patriots didn’t get an insane turnover. The Seahawks ended up snapping the ball with 26 seconds left and one timeout, and though that exact scenario has resulted in three plays at the end of the game with no incomplete passes or three plays and a spike and four plays, I haven’t found (and in fairness, I haven’t looked—I found all of these through Chase Stuart’s twitter account) the point is that there aren’t instances of a team getting four shots at the end zone with 26 seconds left with only run plays.

The question here seems to be whether or not it’s best to maximize the success rate of the first play or maximize the number of opportunities for success. I think we can hold to the idea that, for the most part, a 26-second time limit and one timeout will generally allow for two run plays or one pass play and two run plays.

You see how this begins to look obvious. Either you get to run the ball twice, or you get to run the ball twice and pass it once. Let’s look at why that is.

It is difficult to find data for this—out of the 19 times this situation has appeared since 1999, teams have passed the ball 11 times and run it 8 times. Every time a timeout was used, an average of six seconds ran off the clock. Without a timeout, an average of seventeen seconds ran off the clock. Given that Seattle was having difficulty calling in plays (they took a timeout earlier because of it), getting rid of the short, six-second outlier would mean 20 seconds.

That difficulty in calling in plays doesn’t just mean a delay between the play on second down and the play on third down, but also the subsequent play. On average, it seems you only have two shots, even with a timeout if you run the ball twice with 26 seconds left.

The issue they had with that in general leads me to believe they preferred play maximizing, and so it was an assumption they were working with. After all, for a team that was explicitly concerned with maximizing the number of plays, they certainly took a lot of damn time setting up.

For the purposes of the thought experiment, let’s assume that the fumble rate and interception rate changes near the goal line that happen to the rest of the league apply to the Seahawks. That is, if interception rates increase by a half a standard deviation league-wide, then they will for the Seahawks. Same for fumbles.

Russell Wilson’s depth-adjusted interception rate (that is, adjusting for his average depth of target, his interception rate becomes) 1.6 percent or 1.1 standard deviations below league average. The team-weighted sample for passes from 1-3 yards from the goal line produces an interception rate just below league average by 0.3 standard deviations. Wilson’s expected interception rate in that situation is 1.4 percent.

Doing the same thing for touchdowns produces an expected touchdown rate of 47.9 percent (these happen to be somewhat close to his averages in similar situations, though with a low sample).

For fumbles, the rate of fumble drops to 1.5 percent from 1.7 percent, though one must account for fumble recovery rates. Let’s assume the Seahawk’s fumble recovery rate is 50 percent, because that is a statistic highly susceptible to regression to the mean. So, their chance of turnover is 0.75 percent. We’ll discuss Marshawn Lynch’s success rate below.

Is the change in turnover rate (a 0.65 percent chance swing) worth the extra play? Intuitively, I think so, but we can model it.

Modeling it with a decision tree is easiest. Simulations may be the best approach, but a decision tree shouldn’t be too far off, and if all attempts are independent (I think they are), it should be appropriate.

The first thing to figure out is the rate of success and failure for the Seahawks running with Marshawn Lynch. Lynch has a reputation for being a hard-nosed grinder that is particularly good at grabbing yards in short situations. I think the reputation is fair, but it’s largely based off of his running style, not data. His offensive line happens to be very bad, and that talent mismatch against defensive lines will be particularly exploited in short-yardage situations given how teams approach that scenario.

Aside from the fact that Lynch ran it from the one yard line five times this year and scored on one of them (we can throw it out because of low sample size), in the last three years he’s run it with one yard to go and converted 69% of the time, which is around league average in that time (68.7%). That sample is still pretty small (29 runs), but when you expand it to 1-2 yards, he’s average and at 1-3 yards he’s average (all within a percent point of league average). That may be more a function of play design and OL and not a result of his specific short-yardage ability, but that’s what you’re working with.

The Seahawks offensive line has changed of course; Breno Giacomini is gone and Justin Britt is in, but Britt has not performed better enough for me to consider that change to be significant to this analysis—at least not enough to prefer more applicable small sample data against larger more general data. J.R. Sweezy and Paul McQuistan feel similarly interchangeable in this respect.

For what it’s worth, their Pro Football Focus scores in run blocking (Giacomini was +1.7 in 2012, Britt +0.7 in 2014 while Sweezy was +1.1 in 2014, McQuistan -4.4) are not so stark that I would change the data. Neither are their “Marshawn Lynch running behind them yards-per-carry” with Lynch running 4.5 and 5.1 running behind the right guard and right tackle in 2012 and 4.9 and 4.7 in 2014.

Additionally, if the PFF scores are that important, the run blocking score total in 2012 (+46.3) was higher than in (-25.4). If anything, expanding the data to more years is generous to the idea of running.

For now, let’s assume Lynch is league average in success rate (we’re concerned about successes, not volume of yards) in these situations, which is 68.7 percent of the time.

Seattle loses the game about 9.2 percent of the time and wins the other 90.8 percent of the time. Sounds good.

As for passing and running, it gets a bit more complicated, because the order of things may change the outcomes. But for any combination of running and passing, the maximizing step is to pass first. As Brian Burke at Advanced Football Analytics states (and his model disagreed with the decision to pass):

It was 2nd and goal from the 1 with 26 seconds at the snap. SEA had 1 timeout. If SEA runs the ball on 2nd down and fails to score, that means they must call their final timeout to save time for 3rd and 4th down. And that means they must pass on 3rd down for there to be time for a 4th down.

Passing on 2nd down allows SEA the option of running on 3rd down if necessary, and forces the NE defense to respect both options on all potential remaining downs.

Lynch is a beast, no doubt. He was born for that moment. And he would make a great decoy on 2nd down. It was simply an excellent play by NE’s Malcolm Butler, and it was the one outcome that could backfire on SEA’s decision to pass.

This is pretty important. The threat to run or pass can open up significant opportunities—it’s why 2nd and 1 is the most dangerous down and distance in football (… generally speaking, not this specific situation):

It must give defensive coordinators across the league nightmares. An offense can do anything on 2nd and 1. It can run, and probably pick up the first down, but they could just as easily take a shot down the field without much risk. The QB has the luxury of a no-pressure down. He doesn’t have any need to force the pass and can throw it away if needed. An incompletion still leaves a very manageable 3rd and 1. And failing that, an attempt on 4th and 1 may not be out of the question (especially if it’s a short 1).

A look at the expected points for 1st and 10 situations shows there is statistical evidence that a 9 yard gain is actually preferable than a 10-yard gain. The graph below shows the average expected points for most 1st and 10 plays in the 1st half of all regular season games from 2000 to 2007. (I excluded plays inside field goal range (the 35) and plays within 2 minutes of halftime. This limits the data to normal football situations, when teams are neither desperate nor nursing big leads, and when time is not a consideration. It also removes any bias in the data due to having the option to play very conservatively inside FG range.)


We can see a fairly clear drop in expected points from a 9-yard gain to a 10-yard gain from 2.3 points to 1.6. That’s about a 0.7-point drop in the average number of points scored between having a 2nd and 1, and actually getting the first down. It may not sound like a lot of points, but it’s a relatively large difference for a single yard on a single play.

There is, however, some noise in the data. So how can we be sure that the sudden discontinuity between 9- and 10-yard gains isn’t just a very large random blip?

First, the blip is fairly large. In fact, it’s the largest jump between any two yardage gains. Second, it goes in the opposite direction we’d expect it to go. Further, the graph indicates that on 1st down, a 9-yard gain is not only better than a 10-yard gain, but it’s better than anything up to a 16-yard gain. It also indicates that a 2nd and 3 is notionally just as good as converting 1st and 10. Lastly, we have a good theoretical basis as to why we’d see such a result.

Given the swing that genuine uncertainty seems to cause (half a point from virtually identical field positions), being forced to pass is not the ideal pass-run mix, so it’s better to pass early rather than late (and yes, there’s the issue of reverse-reverse-reverse-reverse psychology but I won’t get into it because the Patriots were expecting a run for the most part and this isn’t relevant).

So the four-play run is 1) run (already a given, they are at second and one) 2) pass 3) run 4) run.

In this case, the Seahawks are successful 93.5 percent of the time and fail 6.5 percent of the time. It’s nearly a three-point swing in percentages, but I think most coaches would take an increase in winning probability by three percent.

More importantly, it demonstrates that it wasn’t obvious to run in that situation.

I mean, there’s a lot of context here that I’m ignoring, but my feeling is that the context is as equally favorable as it is damning. For example, the Seahawks this year with Lynch have been successful on one-yard runs this year (first in the league) while the Patriots were dead last in those situations. Unfortunately, expanding the sample to beyond this year moves Lynch and the Patriots to league average, and I would rather do that then take a low-sample statistic.

If recency bias is to be entertained (and there’s a lot of reason for it), Lynch in specifically goal line situations (which I think we can agree are different—3rd and 1 tends to be defended by 4-3 fronts or 3-4 fronts while goal line situations tend to be 6-2 or 5-3, with a box safety) in the past five attempts has been negative twice, stuffed for no gain twice and positive once.

That’s also only one side of the story. This year, no interception was thrown, in the playoffs or in regular season play, by any team against any team, from the one yard line. Generally speaking this year, runs do worse than passes. That includes Russell Wilson, obviously, who at that point had completed one of his two pass attempts for a touchdown, with the other being a one-yard sack.

Besides, Russell Wilson just threw a touchdown in the third quarter from the three-yard line against man coverage (on second down, no less). In fairness, the first touchdown of the game was also from three yards out and it was Marshawn Lynch. In third or fourth and short situations in the game, Marshawn was 0-1 from the one-yard line and 1-3 from within the two-yard line.

If we’re going to talk about Marshawn Lynch in 3rd and 1 or 3rd and 2 situations specifically this year (as most people are doing by using Football Outsiders’ “power situations” statistic) without controlling for goal line situations, why not Russell? He got a first down 52.9 percent of the time in those situations (17 of them) and threw 0 picks. If you include his scrambling on those plays, the number rises to 55 percent with one fumble (recovered by him).

Forwarding the argument that passing on third down is easier away from the goal line intentionally forgets that that is true for running as well, and the goal line carries Marshawn has had lately have not been good.

Some have referenced that New England has not been great on defense in “power” situations against the run (dead last, according to Football Outsiders) and that that should have guided the thinking. Again, this is almost always a 3-4/4-3 situation and not a 5-3/6-2 situation so again, I’m not sure how it applies, but it’s disingenuine not to look at New England in those situations against the pass, where they are in the bottom of the league in yards per attempt given up in third- and fourth-and-short situations and rank in the bottom third in first down percentage as well.

At the goal line, they rank as the fourth-worst team in touchdowns given up in against the pass, too.

It turns out that New England had disadvantages against both the run and the pass, and looking at specific power situations which don’t account for goal line changes don’t favor the run or the pass.

If we assume that really small-sample stuff about Lynch and New England is relevant, then the assumptions from this year produce the following success rates:

  • 88 percent touchdown rate on runs
  • 38 percent touchdown rate on passes
  • 3.6 percent fumble rate (1.8 percent fumble lost rate) on runs
  • 0 percent interception rate on passes
  • 1.9 percent fumble rate (0.1 percent fumble lost rate) on passes (the previous model had such a low fumble rate on passes it was not included)

That produces the following probabilities in run-run vs. pass-run-run:

  • Run-Run produces a success rate of 96.9 percent
  • Pass-Run-Run produces a success rate of 97.1 percent

Passing gives you another play; it’s that simple. Even the least-generous assumptions towards passing produces a “win” for passing the ball, especially because the only data with which to draw a fumble on a passing play was after a completed pass for 11 yards, which was stripped at the boundary by New England (of Kyle Juszczyk) and fell out of bounds. Which is to say, an identical situation at the goal line would have been a touchdown, not a fumble.

This writeup so far has only been about the decision to pass, not the specific play that was called or about the decision Russell Wilson made to throw the ball that he did. To that end, I’ll reference something I’ve written elsewhere about this:

If 11 personnel is out there against that front I think passing from that group is an easy call and the right one (and again, I’m not putting 11 personnel out there initially, but if that is the case, then a pass isn’t bad). Marshawn splitting far left of the QB in shotgun is a pass alert (and his first step takes away the only good running play there, an outside zone to strength) and I don’t know if I would have aligned that way. Perhaps the best pass call was a play-action bootleg with a run option for Wilson even with 26 seconds left (if you feel the need to pass). With a timeout, I don’t urgently feel the need to pass but I understand it.

With that play in mind, Wilson made the right call—the pick worked exactly as designed and Lockette had acres of space in front of him. The defensive back who has to shift over on top of that pick makes a play on the ball 1/100 times, and that play is rarely an interception. Lockette might have wanted to flatten out the route, but I don’t know given the success of the WR pick, the fact that there wasn’t a safety or LB bearing down and the pass was coming out quickly and flattening out could have hurt timing and forced Lockette to jump for the ball.

If you’re concerned about play-action and a slow developing play with that OL (which makes sense both in terms of the OL giving up pass keys and the OL blocking well), a fade may make sense—although one should remember that after the Patriots adjusted and put Browner on Matthews, he didn’t receive a single target for the rest of the game. Aside from Matthews, is there another receiver you trust with the fade? If so, do it I suppose, but fades are low percentage while slants with picks should probably be high percentage.

Browner did lock up Kearse on the pick block, who would have wanted to run out in front of Butler, but for the most part, the Seahawks got the look they wanted.

Now, do you set up in 11 personnel? Initially, the Seahawks had 21 personnel out there right after Lynch’s run and then took fullback Will Tukuafu off and put Ricardo Lockette in. If the Patriots were going to stay in 5-3 with three defensive backs, you have man coverage on the outside 90 percent of the time with a linebacker playing the short zone underneath—and pretty far away from the stacked receivers on the strength of the formation.

A pick play against man coverage is pretty classic, even in the end zone.

Pete Carroll was playing to win the whole game, not the next play. I think that makes sense, too. Look at the Seahawks’ history with under a minute left against opponents who were within a score. Down eight point against the Denver Broncos this year, the Seahawks left them 59 seconds from their 20, where the Broncos led an 80-yard touchdown drive in 41 seconds to tie the game.

In 2012, they gave the Falcons 31 seconds left from their 28 (in the divisional round of the playoffs), where the Falcons won the game on a field goal by driving 41 yards in 23 seconds. 31 yards away from the end zone, it’s hard not to imagine the Falcons at least having a good shot at the end zone three times in the final 13 seconds (the field goal took five seconds).

The 2012 Chicago Bears drove 56 yards in 11 seconds, and took two dummy plays to run out the clock for a game-tying field goal. With 24 total seconds for their drive, the Bears would have had nine seconds to take a shot at the end zone from thirty yards away if they played for the touchdown instead of the field goal.

Since 2012, the Seahawks have left an opponent within one score less than a minute to go with the ball, and three of those times, the opponents have either tied the game or won it.

Hell, in the previous game, the Green Bay Packers got the ball with 1:19 left down three and kicked a field goal to send the game into overtime with 19 seconds to spare.

And that’s just on their end. Tom Brady has had 17 game-winning drives down by a score in the fourth quarter in his career, and five of them were under a minute long (including a 23-second, 65-yard touchdown drive against the Giants). Against Cleveland last year, he led a 30-second drive for a touchdown to win the game. While his percentage of comebacks in comeback situations with less than a minute left is very low, it’s higher than league average.

It is not weird to think of the Seahawks as being wary, even if they were a tad overcautious.

The idea that the Seahawks “threw into the teeth of the defense” is dead wrong because the only defender that could possibly cover Lockette was was Malcolm Butler, and he was out of position at the time of the throw.

You can call the right play and have the right players, but sometimes someone on the other team makes an amazing play, and that’s not an indictment of you.

Besides, if you practice a goal line package all season, there’s no reason not to use it. Sometimes that package has passes in it, like for any NFL team.

There’s a lot of potential thought here in terms of play design that I don’t discuss—teams are more comfortable at spreading out the defense and running it. If the Seahawks had wider splits for their receivers, would they have given themselves the ability to play a run-pass option? This may have been the perfect example for the quadruple option that Auburn ran this year. The Seahawks openly use Auburn concepts (like the POP pass), so if Carroll prefers bold to conventional, this is perfect.

That’s not a lock either; if you’re New England and you see Seattle splitting wide while you’re in 5-3, I’m not sure it behooves you to split wide—especially if Lynch is lined up closer to Wilson than he was in the actual play that was run. You may just trust your cornerbacks to beat their receivers. After all, it IS a mismatch (just like the New England defensive line and linebackers are a mismatch against the poor Seattle offensive line).

In all honesty, a run-pass option was probably the best play because it allows your linemen to run block, keying the defenders in the middle (all 8 of them) to run up, while still preserving the timing of a one-step drop (at least out of shotgun) and running a pick play on the slant.

But I’m not sure the slant was a bad call by itself (people who regularly bemoaned fade passes decided it was an awesome option suddenly) or that passing was a bad call. I think the play could have used more deception, but at the very end of the day, end-game management is much more complicated than we make it out to be.

At the very least, the suggestion that on average a team is better served passing than running in that situation means the Seahawks deserve more credit for doing the same, especially because the unique circumstances people assign to the Seahawks (Marshawn Lynch is good in short-yardage) isn’t necessarily true.

Any argument that functionally states “but you have Marshawn Lynch” is incomplete because you also have that terrible offensive line and Russell Wilson.

There are some other smart takes around the internet on why it was actually a more defensible call than you might think. Aside from the AFA look, there’s one piece by fivethirtyeight and one by MMQB.

Mike Zimmer, who has heard and deserves his fair share of criticism for his end-game management this year, deserves the same second look when possible.

Share: