Señor Snappy
Mar 3, 20235 min
Updated: Mar 9, 2023
An explosive NFLPA team report card was published today. Let's talk about what it is, before weighing in on what it might mean.
Employee dissatisfaction is expensive, and can express itself in everything from low productivity to outright sabotage. To spot dissatisfaction, most top organizations regularly survey their employees.
Opinion research is easy to do badly. Bad study design is among the "usual suspects" behind all popular misconceptions, from policy to medicine to rocket propulsion. The brain tends to look for shortcuts to preserve energy and get the owner to the good stuff faster, making it keenly vulnerable to manipulation. Bad surveys manipulate the participant to respond in the way the surveyor desires or expects, then sells the output on a pastiche of scientific credibility.
On the positive side, the ability to spot potential flaws in survey results is a superpower and key way to predict the future: erroneous conclusions influenced by a flawed survey will eventually need to be unraveled. One only need assess the importance of a given conclusion to those who benefit from it to chart where the whole thing will end up.
That brings us to the NFL. Does this latest NFLPA "report card" (published March 2023) provide a window into why certain teams succeed and others fail?
In this case, we have a survey of workers conducted by a union about the management. Bias is thus built right in to the survey, before the first question is even asked: the surveyor and the participants already have a relationship built on an agenda which is--by design--hostile to the subject.
Second, survey participants often self-select for negatively-biased; people with a gripe are more likely to respond to surveys. But let's dig in to see what we are dealing with:
The NFLPA survey (per their own description) show both positive and questionable design:
The NFLPA Player Team Report Cards were built on a survey designed to gather feedback from NFL players about their experiences with their team, structured into several categories including coaching, training, equipment, and medical staff.
The survey questions were "crafted to be clear and concise, and avoid leading language."
For example, the coaching category includes questions such as "How would you rate your position coach's ability to prepare you to play in games?" and "How would you rate your head coach's ability to lead and motivate the team?"
The survey also includes open-ended questions to allow respondents to provide more detailed feedback. This approach ensures that the data collected is comprehensive and nuanced. It also allows for interpretation by the surveyor: it can be hard to capture flavor and facts in the same exercise.
The survey was tested with a small group of players to identify any issues with the survey design or questions before being distributed widely. The NFLPA also monitored the response rate to ensure that it was sufficient to draw accurate conclusions from the data. The survey response rate was 1300 (out of an active membership of 2000)--on the surface a strong sample population.
In general, there are several potential weaknesses that can arise in survey design and implementation, including:
Selection bias: This occurs when the survey sample is not representative of the population being studied, which can result in inaccurate or biased data.
Response bias: This occurs when respondents do not provide accurate or honest answers, which can result in inaccurate data.
Leading questions: These types of questions can bias the respondents' answers and lead to inaccurate data.
Bad question wording: Poorly worded questions can lead to confusion or misunderstandings, resulting in inaccurate data.
Non-response bias: This occurs when a large portion of the survey sample does not respond, which can result in a biased sample and inaccurate data.
Limited response options: When survey respondents are not given enough response options, it can result in inaccurate data as their true opinion may not be captured by the available responses.
Self-selection bias: This occurs when participants self-select to take the survey, which can result in a biased sample.
In the case of the NFLPA Player Team Report Cards, there are a few potential weaknesses that may impact the accuracy and reliability of the data collected.
One potential weakness is that the survey is only distributed to active NFL players who are members of the NFLPA. This could lead to selection bias, as the experiences of non-unionized players, retired players, or players who have been cut from their team may not be represented in the data. This could limit the overall generalizability of the survey results. Without knowing who was surveyed, or even any real statistics about them (such as length of time with the team), the survey results are questionable.
Another potential weakness is the potential for response bias. Players may be more likely to provide negative feedback about their team anonymously through the survey, rather than expressing their concerns publicly or to their team management. This could result in an overrepresentation of negative feedback, which could skew the overall results of the survey.
Lastly, while the open-ended questions allow for more detailed feedback, they also introduce the potential for subjectivity and interpretation. This could lead to difficulties in quantifying and comparing responses across teams and categories. It's a trap!
Despite these potential weaknesses, the NFLPA has taken steps to mitigate their impact, such as pretesting the survey with a small sample of players and monitoring the response rate.
I take the NFLPA at their word when they say they "work to encourage all eligible players to participate in the survey" to minimize non-response bias.
Other potential limitations of the NFLPA Player Team Report Cards survey include:
Social desirability bias: This occurs when respondents answer questions in a way that they think will be viewed favorably by others. In the context of the NFLPA survey, players may be hesitant to provide negative feedback about their team or coaching staff for fear of repercussions.
Recall bias: This occurs when respondents have difficulty accurately recalling past events or experiences, which could result in inaccurate data. In the case of the NFLPA survey, players may have trouble accurately recalling specific incidents or experiences throughout the season.
Small sample size: While the NFLPA survey reaches a large number of NFL players, the sample size for each team may be relatively small. This could limit the statistical power of the data and make it more difficult to draw accurate conclusions.
Limited scope: While the survey covers several important categories, there may be other aspects of the player experience that are not captured in the survey questions. This could limit the overall comprehensiveness of the data.
Time frame: The NFLPA survey is administered annually, which means that it captures data from a specific time frame (i.e., one NFL season). This could limit the ability to track changes or trends over time, or to identify short-term issues that may not be reflected in the data.
TL:DR always question data, especially when the source has an agenda. Identify bias built in to surveys and win big!
One last thing: please share this article if you like our content or just want to support the Rivers Lake Yacht Club mission of helping the Chargers gain a home field advantage in Southern California.
We'd also love it if you'd subscribe to this webpage (click the "Become a Member" button at the top of the window), follow us on:
Twitter: @riverslakeyc
Facebook: www.facebook.com/riverslakeyachtclub
Instagram: www.instagram.com/riverslakeyachtclub
Bolt Up!