Monday, May 13, 2013

Lies, damned lies, and statistics.

I was having a conversation with someone the other day about statistics, and their relationship to the draft, when it dawned on me that I might have sounded a bit too dismissive of the film study portion of talent evaluation.  That was probably a mistake on my part. so I thought I would clarify things a bit more.

Among the draft geek crowd, there is a bit of a divide between the guys who examine the numbers, and the guys who feel that it all comes down to studying the film.  Things sometimes get a bit heated (as much as they can between geeks), and insulting comments are thrown around regarding the opposing groups viewpoints.  In general, both sides come out looking rather badly.  Everybody wants to be perceived as some sort of guru, and to show that the other side is composed of morons.  It's sort of a very lame version of the Middle East peace talks, but with more spreadsheets and a slightly lower body count.

Since I would probably get lumped in with the stat geeks, more often than not, I thought I would poke some holes in their balloon first.  The men of numbers tend to be very dismissive of any subjective analysis.  Sometimes this is a good thing, but occasionally they get carried away with things.  If a player's traits can't be measured, they must be ignored.  Only the quantifiable matters.  Except for when the numbers come out wrong, and then they have to come up with new statistics to explain the shortcomings of the old ones.

If there is one issue that I think screws up the statistics crowd, it is probably an over-reliance on regression analysis, and the scatter-plots that come with it.  It's not that this approach has no value.  It's just that people get carried away with it.  They try to find correlations between certain attributes, and an arbitrary measure of success, and forget to question whether this attribute has any sort of sensible relationship to the game of football.  Why does a defensive ends' 3-cone time have a 0.05 correlation to his CarAV (I'm just making this up as an example)?  There can be a reason for this, or it can mean nothing.  If you can't envision how it translates to what a player is doing on the field, it becomes a bit sketchy.  This becomes an issue before we even get to the question of whether CarAV is an adequate measure of success.  Cameron Wake has a CarAV of  37, and is one of the league's premier pass rushers.  Tyler Brayton was a mediocrity, with a CarAV of 39.  Something doesn't add up there.  It ends up being Sex Panther statistics.

It's not that this sort of analysis is bad; it's just that it has some limitations that people sometimes overlook.  In baseball, questions of sample size are less problematic (more games played, more at bats, etc.).  Plus, the situations that players are put in are much more similar, at least when they are batting.  They all face the same pitchers (for the most part), and they have comparable and numerous opportunities.  Does Denarius Moore, playing receiver with the Raiders, have a similar opportunity to Greg Jennings, playing with the Packers?  This is before we even come to the question of whether the best players are getting on the field in the first place.  See Cameron Wake's four years in the CFL, when no NFL team wanted him, as an example.  At some point, expecting the results to be in any way linear, becomes ridiculous.  Maybe it is a product of our childhood education involving coloring books, that makes our obsession with "staying inside the lines" so powerful.  Regardless, I suspect we could find a high correlation between a stat geek's sense of superiority and his proximity to someone who doesn't know how to use Excel spreadsheets.

Really, though, how complicated are spreadsheets?  I could teach a five year old how to do a regression analysis in five minutes (true, a sixty year old might take longer).  Despite my affection for spreadsheets, I have to admit that only people with OCD are likely to spend much time using them.  Spreadsheets do a good job of tidying things up, and making us look more clever than we are, and for that I am grateful.  They don't make us geniuses.

So, now we come to the other side of the table, the "film study" boys.  They generally dismiss statistics, in favor of a pure observational approach.  By eschewing statistics, they also insulate themselves from any analysis that might show what percentage of the time their eyes were malfunctioning.  It's sort of like my view on going to the doctor.  If I never go in for a check up, I am guaranteed to die of natural, or at the very least undiagnosed, causes.  That certainly sounds preferable to finding out about a tumor in my colon.  "Here lies Sid, his ass killed him."  That will never be on my headstone, so long as I avoid doctors.  People will just say,"We don't know what killed him.  It was just his time."

Despite my occasional prodding of the film study crowd, I think there are some areas where they are correct.  No matter what my computer is telling me about a particular prospect, it seems sensible to watch them play at some point.  My only gripe with the film guys, is their short term memory.  This year's 'waist bender with a bad bubble', can be next year's All Pro.  It's good to keep track of how often we turn out to be wrong, so that we can learn from our mistakes.  Everyone wants to point to a player like Vernon Gholston, and say they knew he was going to be a bust, or that they could have spotted Tom Brady.  If you don't account for all the times you were wrong, however, you aren't telling the whole story.

The film study crowd will often say that talent evaluation is more art than science.  Based on  my experience, guru types who make such statements are generally to be avoided.  Yes, there are things that can be difficult to quantify, and some people might indeed have a good eye for talent.  However, people who make these statements are kind of setting themselves up on a theoretically unassailable position.  I can't debate with someone on their subjective views, nor can I prove them wrong.  They might be right, but making mystical claims of insight, is problematic for me.. 

Quite frequently, watching game film does give you an interesting and valuable perspective.  On the one hand, I remember watching Melvin Ingram (DE/OLB, taken with the 18th pick in 2012, by the Chargers), and being really excited about him.  Unfortunately, the computer wasn't quite as sold on him as a prospect, so I was forced to demote him in favor of safer prospects.  On the other hand, I also remember watching Hakeem Nicks play in college (WR, taken with the 28th pick in 2009, by the Giants), and also being  fan.  In this case, the computer liked Hakeem quite a bit.   What really sealed the deal though was watching this particular play.


If you only watched the beginning of this clip, do yourself  favor and watch the whole thing.  The slow motion replay is what you want to see.  Now maybe I made too big of a deal out of this play, back in 2009, but it stuck with me.  I remember thinking,"My god, how did he maintain control of that ball?"  It seemed to show a lot of concentration.  Someone else might suggest that is was just a fluke, or that he got lucky.  To me, at the time, it was just amazing.  I enjoy seeing those "Oh shit, did you see that?" moments, and put a lot of stock in them.  If a player can do something like that, even just once, there is reason to hope that they can do it again....and hopefully again, and again.

When both perspectives, the statistical and the observed, are combined, I think you will get a better view of things, than if you are just an adherent  to one side of the argument.  These two opposed groups are the peanut butter and chocolate of analysis.  They work better together.  Or, since peanut butter and chocolate are actually quite pleasing on their own, maybe these two groups are more like vinegar and oil.  Individually they are repulsive, but capable of making a decent salad dressing when combined.  Still, for what I will write in this blog, I will generally try to refrain from expressing my own subjective views.  There are already enough people out there who will give you subjective opinions, and a select few of them are worth listening to (not he bozos on ESPN).  For the most part, I will just give you the numbers, and you can go watch the players yourself, to form your own conclusions.  There's no reason to think that your impressions of a player would be any worse than mine.

No comments:

Post a Comment