The Inconvenient Truth about Survey Bias

Today I want to talk about a survey that's been making the rounds1, put together by a 13-year-old named Victoria who's trying to gather evidence about the experiences of transgender youth and their families. Now, I appreciate Victoria's initiative and desire to contribute to an important conversation, but we need to be real careful about how we gather, interpret, and use this kind of data.

See, the thing about surveys is that they're only as good as the people you ask. If you're only asking folks who already agree with you or who have a particular axe to grind, then you're not getting a representative sample. You're getting a skewed picture that confirms what you already believe. And that's exactly what's happening with Victoria's survey.

The respondents were all recruited from a group called PITTparents.com. Let's be clear: this group is not a neutral or unbiased source of information. They actively promote a gender-critical ideology that seeks to undermine the rights and identities of transgender people. They spread misinformation about gender-affirming care and perpetuate harmful stereotypes about transgender youth being "confused" or "misled" by social media and peer pressure. They often advocate or encourage harmful and abusive practices when dealing with trans kids.

By only surveying people from this one group, Victoria is not getting a balanced or representative view. She's getting a cherry-picked sample that's going to give her the answers that align with PITT's agenda. It's like if I wanted to prove that the earth is flat, so I only asked people at a flat earth convention. Of course, I'm going to get results that favor a flat earth, but that doesn't make it true.

Now, I don't want to come down too hard on Victoria. She's young, and it's clear that she cares about this issue. But caring isn't enough. We have to be willing to question our assumptions, to look at the data objectively, and to follow where the evidence leads, even if it challenges our preconceptions. And to her credit, when one person gently pointed out to her that she has some serious flaws and gaps, she was thankful for the feedback and said she would do better next time. Let’s hope so, Victoria, because I would be so happy to see what the next survey is and what the results look like.

So, with that said, let’s break down the issues with Victoria’s study:

Sampling and Survivorship Bias

1. Sampling Bias: The survey was conducted among members of PITTparents.com, likely meaning the respondents share similar views. This can lead to sampling bias because the group may not represent the broader population of parents with transgender children. For example, parents who are more skeptical of gender-affirming care might be overrepresented.
2. Survivorship Bias: The survey might only capture the experiences of parents active in specific online communities, excluding those with positive or neutral experiences. This can lead to a skewed perspective that emphasizes negative outcomes.
3. Homogeneous Group: PITTparents.com likely attracts a specific group of parents who share similar views or experiences regarding transgender issues, resulting in a homogeneous sample.
4. Self-Selection Bias: Parents who chose to participate may have stronger opinions or more negative experiences, skewing the results. Those with neutral or positive experiences might be underrepresented.

Confirmation Bias and Lack of Control Group

1. Confirmation Bias: The goal to find evidence of harm caused by gender-affirming care can unintentionally influence how questions are framed and responses interpreted. It's important for research to remain neutral and open to all possible outcomes.
2. Lack of Control Group: The survey does not include a control group of parents who are not part of PITTparents.com but have trans kids or hold different views on transgender care. Without this, it's impossible to compare and contrast experiences holistically, leading to skewed results.

Correlation vs. Causation

• Linked but Not Caused: The survey results suggest correlations, such as between social media use and identifying as transgender. However, correlation does not imply causation. Many factors could influence these outcomes that the survey does not account for. We should not draw conclusions that the survey cannot support.

Logical Fallacies

1. Hasty Generalization: Drawing broad conclusions from a small, non-representative sample can lead to hasty generalizations. For instance, concluding that all transgender youth are influenced by social media based on your survey results.
2. Post Hoc Ergo Propter Hoc: Assuming that because one event (identifying as transgender) follows another (having a trans friend), the first event caused the second. This is a fallacy without further evidence.

Recognizing Bias in Data Interpretation

To understand how bias can shape perceptions, let's consider how a parent might be influenced by skewed information. Imagine a parent comes to a forum like PITTparents.com, seeking advice about their child who has recently come out as transgender. They're confused, scared, and looking for information and guidance.

In response, they might be given a very specific narrative that their child has been "brainwashed" by a cult-like transgender movement, and that affirming their child's identity is harmful. If this parent expresses doubt or uncertainty about this narrative, they may be met with anecdotal horror stories, cherry-picked statistics, and appeals to emotion designed to overwhelm their critical thinking and reinforce a gender-critical worldview. Surveys like this one would seem to lend a lot of weight as evidence to support these narratives, adding a new foot soldier to the echo chamber.

What Can We Learn From This?

Victoria's survey teaches us the importance of critically evaluating the sources of our data and the methodologies used to collect it. It highlights the need for diversity in our sample groups and the importance of neutrality in our questions. By understanding these biases and fallacies, we can better identify misrepresentation and avoid drawing unsupported conclusions. Cultivating this awareness in ourselves (and each other) helps us ensure that our research is robust, reliable, and truly reflective of the experiences and needs within the transgender community.

So, at the end of the day, we all want what's best for these kids. But we're not going to get there by relying on biased data or promoting harmful ideologies. We need real, robust research that looks at the big picture and takes into account the diversity of experiences and needs within the transgender community.

So let's keep asking questions, keep seeking answers, and keep trying to reach those who are lost to the clutches of this “gender-critical” ideology, but let's do it in a way that respects the humanity of everyone involved. Because that's the only way we're going to make progress and build a world where every kid can thrive as their authentic self, free from discrimination and prejudice. I hope that Victoria, in her next survey or research project, can keep an open mind and find out for herself where the truth really lies.

Thanks for reading, folks. Until the next time I can write something, take care of yourselves and each other.