Tuesday, July 15, 2014

What is a petard anyway?

It seems that I've always had an incorrect understanding of what this word really means.  As it turns out, a petard is basically a bomb made for blowing up gates or doorways.  A soldier would approach a fortification with their petard in hand, light the fuse, and then run like hell.  According to the mighty Wikipedia, "Petard comes from the Middle French peter, to break wind, from pet expulsion of intestinal gas".  How awesome is that?  We basically have a fancy way of referring to a fart bomb.

Unfortunately, things didn't always go according to plan, and we are probably most familiar with this word from Shakespeare's Hamlet, where he referred to the possibility of an individual being "hoisted with his own petard", or launched into the air by a bomb of one's own making, at least in a figurative sense.  Sometimes things just don't go the way you expected them to, and the results can be a bit unpleasant.

With that sort of unexpected result in mind, I made a chart that we named PETARD.  This was intended to show what the Ravens might expect from one of their draft picks, when it comes to the percentage of games started in their first four years in the NFL, depending on when the player was selected.  While individual results will vary, based on numerous factors such as actual ability (though this probably matters less than you would hope), I just wanted to get a sense of what the average result for a player might be.  With this, we can get some rough idea as to what the Ravens' management is expecting from a player, and how close a player came to meeting these expectations.

While many people may try to grade a team's draft class, based on their own subjective opinions, the degree to which we should pay attention to any of this jibber-jabber is highly debatable.  Like many other people, I have my own views on these things, but there's no reason for anyone to take the opinion of a dilettante of NFL statistics, such as myself, too seriously.  In the end, it is really just a question of whether the players appear to be meeting the team's expectations, which I want to examine.

So, what I decided to do is to calculate the total number of games we should expect a player to start in their first four years, and find the cumulative total for each draft class, and then see how close the entire class actually came to this projected average result.  Whether the player is arguably any good, really doesn't matter to me.  If the team wants to start a player, despite that player performing horrendously, is entirely up to them.  Take Gino Gradkowski, for example.  According to PETARD, Gradkowski would have been projected to start 17.5 games in his first 4 years, based on where he was drafted.  So far, he has started 16 games, performing quite horribly according to most people's accounts.  If the Ravens choose to continue giving him starts, that is up to them, though it seems unlikely that this will happen.

If, in a given year, PETARD projects that a draft class should generate a hypothetical 140 total starts over a four year span, we are simply seeing how close that class came to meeting this expectation.  If a draft class accumulates the 140 projected games started, PETARD would give this a result of 100% efficiency.  This wouldn't necessarily mean that a draft class was good, it merely means that a particular draft class fully reached the average expectations the team might have had for it.  The total number of projected starts will obviously vary from year to year.

The number of draft picks that a team has in a given year, and how high those draft picks happen to be, really makes very little difference.  A team could just as easily have one draft pick in the 7th round, and still be judged to be quite respectable in the eyes of PETARD.  For example, if this solitary and imaginary late round prospect was chosen with the 238th pick, we would only expect this player to start 4.35% of all games within their first four years, or just 2.78 games in total.  If this imaginary player started this insignificant number of games, PETARD would still view this draft class as having met expectations, and give it a grade of 100%.  If a draft class produced more starts than expected, the grade could easily rise above the 100% mark.

Now, since the Ravens are frequently viewed as a team that does quite well in the draft, I wanted to see how they have done over the course of time in meeting what appear to be their own expectations.  Since much of their reputation seems to stem from their notable successes from the late '90s to the mid 2000s, that was the period of time I chose to target for all of this.  The chart below shows how PETARD would evaluate their performance from the years 1999-2011.  The goals isn't to judge the Ravens, but instead, to let them judge themselves.  Basically, this is sort of like sending the Ravens' management to a Montessori school.  We can just sit back, and let the team hoist themselves on their own petard.

For the most part, you can just ignore the trendlines that are on the chart, as they are just there for a later discussion, which I may or may not end up pursuing.

Before we get much further into this, there is a minor adjustment I think we should make.  When we initially came up with our calculations for PETARD, we excluded a small handful of kickers, punters and fullbacks, since the expectations are clearly quite different for people who play at these positions.  In the above chart, you will see that the Ravens' result for the 2007 draft class is a rather insane 142%, which would suggest that they performed 42% above what we would consider to be the average expectation.  Unfortunately, a lot of that is driven by the inclusion of fullback Le'Ron McClain, who was selected in the 4th round with the 137th overall pick, which is about as high as you would normally expect a fullback to ever be taken.  Normally, we would only expect a player to start about 18.6% of all games in their first four years, if selected at that position, while McClain actually started 84.375% of all games.

Since the inclusion of such a strange player, at such an arguably peculiar position,  produces rather disturbing results, I thought I would also include an alternative chart for this same time period, that excludes Le'Ron McClain from the equation.  I think this might present a more sensible picture of the situation, but you can make up your own mind about this issue.

The Ravens have had their ups and downs, just like any team, but it does seem to be quite clear that from 1999 to 2007 they almost always managed to draft players who succeeded in meeting their expectations, at least in a cumulative sense.  This doesn't mean that I fully endorse all of the decisions the team made during this period of time, but they certainly appear to have been at their best during this period.  The only major blip on their radar was their horrific performance during the 2004 draft, where their selections produced practically nothing while under their rookie contracts.  On the other hand, in 6 out of these 9 years their draft classes fully met expectations, and frequently exceed them by a rather good margin.  The average result during this period would have been about 103.5%.

Sadly, once we start looking at the years after 2007, things seem to have hit a bump in the road.  From that point forward, the team seems to have consistently fallen a bit short of the mark, with their best results only managing to measure up with what might have been viewed as a rather disappointing result in their better days, with an average outcome of around 84.8%, during the 2008-2011 period.

Technically, I really shouldn't be including the 2011 draft class in any of this, as they haven't completed their fourth year yet.  Still, it's fairly simple after three years to project things forward a bit, and see that they too will probably fall a bit short of expectations.  While their result for now is projected to be about 90%, I actually suspect it might end up being perhaps 1-2% higher.  A lot of this projected grade hinges on Jimmy Smith and Torrey Smith starting 32 combined games in the upcoming 2014 season, and injuries can obviously have a huge impact on this.  According to PETARD, the 8 players that the Ravens selected in 2011 were projected to generate 127.6 cumulative games started.  So far, after their first 3 years in the league, they have produced 83 games started.  With just one year remaining, that leaves them 44.6 games below expectation.  Now, if Jimmy Smith and Torrey Smith both start all 16 games this season, that reduces the 2011 draft class' debt to 12.6 games that would still need to be picked up, or 2.1 games started per player, for the six other prospects that were taken in 2011.

That would seem like a fairly easy goal to achieve, but it probably isn't.  Tandon Doss and Anthony Allen aren't even on the team anymore, and seem unlikely to start for their current teams (which Ozzie would still get credit for).  Tyrod Taylor, as a backup quarterback, also would appear unlikely to see any action.  This increases the debt that each of the 3 remaining players would have to make up for from 2.1 games started/player, to 4.2 games started/player.  Considering that this debt falls on the shoulders of Jah Reid, Chykie Brown and Pernell McPhee, I think it is safe to say that the goal will most likely not be reached.

When we consider the 2012 draft class, things don't seem to improve very much.  So far, the bulk of the heavy lifting being done by the 2012 class is coming from Courtney Upshaw and Kelechi Osemele.  While Upshaw has started 68.75% of his games so far, his shortcomings seem to have been a likely factor in the team's decision to sign the veteran pass rusher Elvis Dumervil to split snaps with him.  It's not exactly a ringing endorsement of your top pick when a team is making those sorts of moves after a player has been in the league for just one year.  Still, it's way too early to assign a grade to this class.  Nonetheless, I have some concerns about this group.

I realize that some people won't like the idea of judging a draft class based on how many "starts" it produces relative to a weird idea of "expectation".  All I can really say is that I prefer to keep things fairly simple.  At some point, a team has to find/produce starting caliber players, or else they are likely to run into problems down the road.  There's plenty of room to debate certain aspects of this.  Sometimes a player manages to emerge a bit later in their career, even if they weren't a starter in their first few seasons.  The problem with that is a player's long term contributions probably shouldn't be a deciding factor in evaluating a draft pick.  One of the primary values of a draft pick is the relative affordability of their rookie contracts.  The draft isn't really about acquiring talent.  It is about acquiring cheap talent.  If they're not getting on the field fairly early, a team really isn't getting much value out of their investment.  It also means that the team will have to compensate for this by signing/starting veteran players, who are more likely to be expensive, while also limiting their ability to evaluate whether the youngsters have anything to offer.

People might also point to the Ravens' fairly successful run during the past 5 seasons, where they have frequently managed to make it to the playoffs.  Of course we could argue that a lot of the foundation of those successful teams was probably built during the drafts of the late '90s and early 2000s, with selection like Ray Lewis, Ed Reed, Jonathan Ogden, Marshal Yanda, Haloti Ngata, Terrell Suggs, etc.  As these players get older and retire (which some have already done), does it appear that the team has really found replacements of a similar caliber?  Or, will the team start to decline, as the draftees of the past 7 years are forced to become the new faces of the franchise?  I may end up posting something a bit later that relates to this, but much like the popular opinion you hear on this subject, I do think it generally takes about 3-4 years for the full effects of a draft to be felt, whether they are positive or negative.  If the Ravens are going to go into a decline, I wouldn't be surprised if the consequences of recent drafts only start to show up right about now.  Perhaps this is already occurring based on their relatively poor performance in 2013.

There is one other factor in all of this that I can't dismiss.  There is always the possibility, no matter how unlikely I personally find it to be, that a team can simply shift its philosophy towards giving playing time to young players, which could affect their PETARD results.  So, what happened in 2008, that might explain the team's apparent decline in draft efficiency?  The obvious answer might be the hiring of John Harbaugh, who happened to arrive in 2008, right when PETARD seems to suggest things really went downhill.  Whether Harbaugh is hesitant to start young players, or whether the young players have failed to meet his expectations, is a difficult question to answer.  Either way, the result is the same.  It all ends up amounting to a reduction in the value coming out of the team's draft picks.

While none of this can truly say how the Ravens have performed in comparison to other teams, I do think it points to the possibility that whatever magic or luck guided the team in the past may have begun to fizzle.  Perhaps, much like the way that fans refer to Ozzie Newsome as The Wizard of Oz, he is really just a mere mortal hiding behind a curtain, madly pulling levers in a desperate attempt to keep the citizens of the Emerald City content.  If I only I had a brain, I might be able to answer these questions a bit better, but for now, I remain pessimistic about the team's immediate future.

Tuesday, July 1, 2014

Let's Make Broad Generalizations!

This is only going to cause me trouble.

I'm about to make what is probably a huge mistake, by diving into a subject that I find interesting, but which I suspect is only going to cause me a fair bit of annoyance.  Fortunately, I can be a bit of an idiot, and am not very good at avoiding foolish behavior.  So, here's the question that's currently rattling around in my head.

What should we realistically expect out of a draft pick?  Or, to what degree does being picked highly influence the path of a player's career, for better or for worse?

I realize that a number of people have tinkered around with this sort of question.  A lot of the attention nowadays seems to be directed towards running regressions that compare CarAV(or Approximate Value) to where a player was selected in the draft.  The results of some of these examinations can be interesting....but I just can't fully jump aboard the CarAV train.  I'm not saying that CarAV is a bad statistic, it's just that I have some qualms about committing to it at this point in time.  As these sorts of geeky stats go, I just think it might be trying to do a bit too much.

I think my main problem with CarAV is that many people seem to be using it as a measure of a player's performance or contribution to a team.  Some people seem to be using CarAV as a qualitative measure, when I think it is probably more of a quantitative statistic.  It does a fine job of saying whether a player is getting on the field, but I suspect it doesn't say much about what someone does once they are allowed to play.  I'm just not sure we are at the point where we have a "one ring to rule them all" statistic, to easily sum up such a potentially complex question as to whether a player is "good" or not.  Instead, I think CarAV might be more of a 'perceived performance' or 'perceived contribution' estimate, which is a very different sort of idea.  It might tell us who the coaches have some confidence in, but not whether the coaches are right to feel this way.

A player who is on a particularly effective unit (offensive line, defensive line or whatever) probably has a clear advantage in building up their CarAV results.  It tends to be a very end result oriented statistic.  If your team/unit does well, you get graded well, even if you might have been the weakest link in the chain.  Getting selected for Pro Bowls and All Pro teams also has a significant effect on CarAV, and those selections can often be popularity contests that might favor high draft picks over a comparable player who was drafted later.  The single biggest factor in CarAV is probably a player's ability to simply get on the field in the first place, where high draft picks will always have the edge, almost regardless of their actual performance.  I just feel that CarAV might be weighted with some confirmation bias regarding high draft picks.

Over the first 6 years of his career, Evan Mathis' average annual Approximate Value result was a 2.  That's not a very good result, in the CarAV world.  Opportunity clearly plays a significant role in all of this, as he (a 3rd round pick) was only asked to start in 22 out of a potential 96 (22.91%) games during this time period.  In the last 3 years, when he was finally made a regular starter, his average annual Approximate Value was 8.66, a much more impressive result.  On the other hand, the generally mediocre Michael Oher, who was selected in the 1st round, had an annual average Approximate Value result of 8 in his first five seasons, and his results never dipped below 7 in any year.  Of course, Oher started every single one of the 80 games from the day he was drafted, despite wavering between an average to below average level of play.

Now, some people might argue that perhaps Evan Mathis improved over time, and I'm sure that his coaches would love to take credit for this improvement, but I just can't buy into that idea.  I think there were some fairly solid reasons to suspect that Mathis was always the more gifted player, as you can see in the post on the Lobotomy Line.  Does anybody believe that Michael Oher is a comparable/superior talent to Evan Mathis?  Or, is it just more likely that the challenge of exceeding/altering people's expectations of a player is a more difficult proposition when a team has invested less in a particular player in the first place?  Hope is a flame that constantly burns bright when it comes to high draft picks.  Hell, there are people still waiting for a Tim Tebow resurgence.

Now, I realize that this is just one peculiar example of a possible shortcoming in CarAV, and criticizing CarAV really isn't my goal.  CarAV has its uses.  All I'm trying to say is that there are some issues which cause me to have concerns about whether the best players are consistently being put on the field.  I have little doubt that the majority of high draft picks are reasonably talented, but assuming that the talent is there simply because they are high draft picks is a very different question.

Instead of looking at 'perceived performance' or hype, relative to where a player is selected in the draft, Reilly proposed that we ignore the talent/quality of performance issue altogether.  He was just curious about the degree to which a player's draft position relates to a team's willingness to give someone a starting role on the team.  More specifically, he suggested that we chart the percentage of potential games started during a player's first four years in the league.  A four year period was chosen since, with the exception of some 1st round draft picks, that is the typical length of a rookie contract, which is part of the real issue we're eventually going to try to figure out.  So, without further jibber-jabber, this is basically what the results look like.

We're going logarithmic, like an ALPS Blue Velvet potentiometer!

This chart is based on the Ravens' draft picks from 1996-2010.  Beyond being my home team, and the organization I am most familiar with, they really are a great team to run these experiments on.  Having one GM, Ozzie Newsome, who has overseen such an extensive period of time running the team, eliminates some of the fickle fluctuations that might come with a more volatile organization.  For better or worse, their behavior should be relatively consistent, at least compared to teams with more turnover.

During this period of time 118 players were selected by the Ravens, though we did remove five of them from the discussion.  Since we are only doing this to look at the team's tendencies, behavior and biases, Reilly removed Sergio Kindle and Dan Cody from the list, because they were relatively high draft picks who never played due to injuries that were factors before a single game had even been played.  Remember, we're not trying to judge the team's performance or luck in the draft (not yet, but we will later), just their tendency to give "starts" to higher draft picks.  If a player was never healthy enough to play, the team never gets to make much of a decision in those cases.  We also removed any kickers, punters (Dave Zastudil) or fullbacks (Le'Ron McClain and Ovie Mughelli) who were selected before the 5th round.  These positions generally don't get credited with "starts" in the first place, so they just create a bit of chaos if they are included.  In reality, only Le'Ron McClain would have had a significant effect if he had been allowed to remain in this list, at least compared to Zastudil and Mughelli.  We also obviously can't include results from more recent drafts, because the four year time period for those players hasn't run out yet, though they will eventually be included.

The data could be tightened up and manipulated a little to produce a better R^2 value, but really, 0.5697 is rather adequate for our humble purposes.  I prefer simplicity over tidiness and perfection, since a better data fit could spark criticisms of nudging the results too much.  We're not trying to predict what percentage of potential games started that a player will have in their first 4 seasons.  Instead, we just want to figure out what an average result might be, relative to where a player was selected.  Why we're curious about this, is something we'll get into at a later point in time.  For now, we're just going to refer to the data from this chart as PETARD (because it makes me giggle like a little girl).  Eventually, we'll have to turn this into an amusing acronym to justify naming it this.  It should also be noted that once we get above the 5th pick in the first round, it becomes impossible to meet or exceed expectations, so we actually capped everybody at an average expectation of 100% of games started in their first four years, regardless of how insane that may be.

Like I said though, this isn't really about judging whether a player is good or not.  There are clearly numerous data points, particularly for mid-round players, that wildly diverge from the trendline, so individual outcomes can be fairly unpredictable.  Despite that, it seems reasonable to suspect that players who exceeded these average expectation for where they were selected were possibly doing something right, and vice versa, for players who failed to meet expectations.  You could say that we are simply looking to identify who the outliers are.  In the future we will explore the stories behind these players who produced surprising and unexpected results, in an attempt to better understand them.   I fully realize that many people aren't going to like the idea of using "percentage of potential games started" as a metric for making judgments (we're still working on the Moxie-Meter), but...you can't please everyone.  At the end of the day, I do think being a "starter" is more valuable than not being a starter, so we're not going to over-think this.  This is just meant to be a simple and reasonably objective way of gauging things, and some shortcomings are naturally going to exist in this process.

Really, there's probably not much in this post that should matter to anybody, but some of the issues I've been considering will refer back to this.  It was just easier to throw this out there in advance, rather than trying to fit it in later.  I think the more interesting discussion will come somewhere further down the road.