Are the data collection methods easily available?

How did they obtain survey respondents? This matters. Let’s consider our example claim, and think about places where survey respondents could have been obtained (such as coding bootcamps, social media, meetup groups, users of a specific website, etc.)

Did they obtain a survey population that accurately represents the whole? Again, it’s possible to have a small sample of respondents that fairly represent the whole population, and it’s important to be up front about that.

Did they try to determine some sort of data normalization? Or did they address it if not? For example, a group of engaged coding bootcamp participants are more likely to answer a technical survey than a group of developers in a large enterprise organization.

Who paid for the study? Who performed the study? Do they benefit from a positive outcome?

It’s fine for someone to have paid for it. It’s also okay for them to benefit from a positive outcome. But the study should be up front about these things. Don’t fall for any ”research” that is merely a marketing ploy- lots of those exist. Faux research isn’t research.

Who funded this work? If you can’t figure that out, throw away the entire study. Scientific research is required to say who funded the study in the paper (it’s usually the last section of the paper, or in the footnote if it’s printed in a journal). It usually goes hand-in-hand with a conflict of interest statement. In general, if it’s the most amazing thing you’ve ever read, but you can tell that the person who funded the research will somehow profit from these results, then you should be careful when interpreting the results. Who benefits from this work? It’s easy to tweak any results to reflect what we want it to say.

Who did the research? If the research was meant to be, say, a study about the community at large, but then only has committee members from a specific circle of that same community, assume implicit bias.