Vanity Metrics are Useless

Improving Student-Athlete Surveys: A “How-To” Guide for Athletics Administrators

This is the first in a series of posts that will provide guidance on moving your student-athlete survey from a “vanity metrics” activity to a valuable and actionable tool to gain competitive advantage for your department.  

Post #1: Vanity Metrics are Useless

The single most important improvement you can make to your student-athlete survey, is to move away from asking insignificant questions and gathering useless data.  Many evaluation tools are rooted in “vanity metrics” as opposed to actionable data.  Serial entrepreneur Eric Ries gives a great overview of the difference:

“The only metrics that entrepreneurs should invest energy in collecting are those that help them make decisions. Unfortunately, the majority of data available in off-the-shelf analytics packages are what I call Vanity Metrics. They might make you feel good, but they don’t offer clear guidance for what to do…That makes sense, since they are the easiest to measure and they tend to make you feel good about yourself.”

Survey questions crafted using vanity metrics tend to measure things like satisfaction, hits and likes.  Let’s dissect a common example from a student-athlete survey to illustrate the difference.

Typical survey question focused on vanity metrics:

On a five-point scale, rate your head coach.

Say the head coach gets an average rating of 4.4 from their student-athletes.  Seems pretty good, right?  But what does that really mean?  What indicators are your athletes really rating them on?  What kind of actionable feedback can you give your coach from that question?  What should they do to improve?  What are they already doing well that they shouldn’t change?  And how does that coach compare to other coaches of the same sport at other institutions?

This is a great example of feedback that lacks context and has little meaning.  Layering in “stars”, “likes” or “thumbs up/down” may look weightier but in the end you still get the same useless data.

Survey focusing on actionable metrics:  

Using a five-point scale, please rate your head coach’s communication ability:

My head coach communicates clearly with me.  4.8

I have talked with my coach about my role on the team.  4.7

My head coach uses appropriate language around the team.  4.9

My head coach knows how to successfully develop and communicate a tactical game plan.  3.2

The average for this head coach is actually the same as in the first example – 4.4.  But we now know that by assessing just this one aspect of the head coach – communication - they excel at the day-to-day communication, but struggle when it comes to communicating technical, game-day strategy to their athletes. Now we clearly see that this coach has many strengths that we can build on to improve an area that needs some work.

Additional variables that impact coach ratings:

Now take this actionable data and layer in other variables that impact coach feedback such as:

Were you recruited by your current head coach?

What is your average level of playing time?

Do you have a leadership role on your team?

Suddenly you have a much richer picture of this individual on your staff, and more importantly as the AD you are better equipped to support this individual in their development and enhance team success.

Improving your Student-Athlete Survey:

Crafting an actionable student-athlete survey will yield a myriad of benefits for your department.  Some examples are:

• Better recruiting and retention of student-athletes who are a good fit for the strengths of their coach and overall program culture, thus minimizing transfers and “problem” student-athletes.
• Demonstrate an arc of progress and improvement in your program.  If student-athletes rate their locker-room as substandard, the data (especially when used in comparison with your competitive peers) can support prioritizing resources for a renovation.  
• Educational opportunities for key support staff such as athletic trainers and strength coaches regarding confidentiality, communication, and “appropriateness.”

Your Action Item: Review your current survey questions and determine whether they are focused on vanity metrics or actionable data.  Do your results help you support measurable coach/staff development?  Are you gathering data that supports resource allocation requests for your program to your President/Board of Trustees?  Are you gathering information to help you mitigate risk in your program?  Do your results demonstrate improvement in the student-athlete experience at your institution year over year?

If you are generating pages of results that leave you with little information you can act on, it’s time to invest in crafting a more useful student-athlete survey.

Next Up in Post #2: To Share or Not to Share.  

In an age of ever-present sharing it may be tempting to reveal your student-athlete survey results.  We’ll discuss why selectively sharing your data can put your institution at risk.

Athlete Viewpoint is an actionable tool designed by industry experts to alleviate the time and labor your staff spend on this valuable process.  Let us do all the work gathering feedback, so you can spend your over-stretched minutes improving the student-athlete experience and making data-driven decisions in a strategic and thoughtful manner. AV was co-founded by Michael Cross, a tireless advocate for both the student-athlete experience and the overworked AD.

Watch a brief video, view a sample survey, and schedule a demo of Athlete Viewpoint at www.AthleteViewpoint.com