In the past, I have blogged about stereotype threat and stated my view that the research on stereotype threat is extremely weak. I take a similar position in Chapter 30 of my book, In the Know: Debunking 35 Myths About Human Intelligence (Warne, 2020).

I base this view on the characteristics of stereotype threat research, which shares a lot of methodological problems with research that does not replicate. Some of these are small sample sizes, low statistical power, high flexibility in research design and data analysis, and a strong social incentive to find results supporting the theory. All of these were obviously present in a recent meta-analysis on stereotype threat in females that I have criticized.

But this is all indirect evidence is not as important as the results of direct replications of studies that support the theory. After all, if stereotype threat is a real phenomenon, then it should appear consistently in studies that trigger the threat experience in stereotyped individuals. If the phenomenon does show up in studies that have the same methodology, then it can put the worries about methodological problems to rest.

I used my university library’s OneSearch tool, which simultaneously searches dozens of databases (PsycINFO, PsycARTICLES, Education Full Text, ERIC, the Psychology & Behavioral Sciences Collection, and Sociological Collection), to search for the keywords “stereotype threat replication” (without quotes). I searched the same terms in Google Scholar.

These searches only produced four articles which might qualify as replications of three original studies. Given the hundreds of studies that have been conducted on stereotype threat, this indicates that the vast majority of stereotype threat studies have never been subjected to a close replication (judging by the published record, at least).

Replication 1: Kellow & Jones (2008)

Kellow and Jones (2008) published the earliest replication of a stereotype threat study. However, it gains this distinction if one uses the term “replication” somewhat loosely. The study duplicates their earlier study on stereotype threat in African American high school students (Kellow & Jones, 2005) by using the same materials and procedures as the earlier study. However, the 2008 study used random assignment (whereas the first one was a quasi-experiment) and added dependent variables. Because this would improve the study without fundamentally changing the theory it tests, I still count this as a replication.

The results are clear: African American students had no change in average scores, regardless of whether they were exposed to a standard test administration scenario (the stereotype threat condition) or a testing scenario that stated that there were no race or sex differences in test performance (the non-threat condition). The effect size was a mere d = .01.

Replications 2 and 3: Gibson et al. (2014) and Moon & Roeder (2014)

Gibson et al. (2014) and Moon and Roeder (2014) each conducted an independent replication of Shih et al. (1999) using the same materials and procedures. In the original study, Asian American college students were randomly split into three groups. One group was reminded of stereotypes about Asians being good at math. The second group was reminded of stereotypes about females being bad at math. The third group had no reminders and served as a control.

Neither replication study showed that any statistically significant difference in math test performance across the groups in the full sample. The picture is a little more complex, though. Gibson et al. (2014) did find some differences that barely met the threshold of statistical significance (ps = .02 and .04) across groups after eliminating data from subjects who stated that they were aware of stereotypes about their group. (Shih et al., 1999, did a similar follow-up analysis in the original study.) However, Moon and Roeder (2014) performed the same analysis and failed to find any statistically significant results. Where Gibson et al. (2014) and Moon and Roeder (2014) do agree is that a subject’s personal identification with mathematics is not a relevant moderator for stereotype threat, contrary to Shih et al.’s (1999) original study.

At best, the results of these replications are ambiguous. If stereotype threat does exist in Asian females, it only occurs in a subset of this demographic group, and the p-values would not survive a correction for multiple testing. Even then, one replication shows a total lack of support for any aspect of Shih et al.’s (1999) study and stereotype threat theory.

Replication 4: Finnigan & Corker (2016)

Finnigan and Corker (2016) conducted a close replication of a study by Chalabaev et al. (2012). In the replication, the stereotype threat effect not statically significant, no matter which of six different conditions in the study were compared. The results are shown in Figure 1 and are the nullest null findings I have seen in a long time. This is an unambiguous strike against stereotype threat theory.

Analysis

These researchers subjected stereotype threat to four replication attempts and got three unambiguous failures and one replication with mixed success — if one throws out some of the data. Anyone who thinks that stereotype threat is real must grapple with this poor track record.

What is noticeable about these replications is that the methodology gets better as time goes on. Kellow and Jones’s (2008) study pre-dates the replication crisis, and it shows. Gibson et al. (2014) and Moon and Roeder (2014) pre-registered their replications and made their materials and data open. Finnigan and Corker (2016) did the same and also have a sample size that was over four times larger than the original (and larger than the other replications combined).

I do not think that stereotype threat is real, and I think that researching stereotype threat further is a waste of time and money. But I understand why some people might not be ready to discard the theory. That is why I propose a moratorium on new stereotype threat studies until any stereotype threat findings have been shown to replicate in pre-registered studies with high statistical power. Send in the clones.

As a skeptic of stereotype threat theory, it is frustrating to read these replications. In two of them (Gibson et al., 2014; Kellow & Jones, 2008), the authors engaged in post hoc rationalization in an attempt to salvage the theory and show how their results, despite contradicting the original findings and the predictions of stereotype theory, actually support the theory. The other two groups of authors (Finnigan & Corker, 2016; Moon & Roeder, 2014) avoid this pitfall, but neither suggests that stereotype threat theory may be wrong.

Comparison with a Similar Search For Replications

I am not the first person to conduct this type of analysis. Stoet and Geary (2012) conducted a similar analysis and found 23 studies that they classified as replications of a study of stereotype threat in females conducted by Spencer et al. (1999). Stoet and Geary (2012) found that only 30% of later studies could replicate Spencer et al.’s (1999) finding of stereotype threat in females.

Stoet and Geary (2012) found many more “replications” because their definition was broader than mine. The four studies I found were explicitly planned as replications by their authors, and they worked to adhere as closely as possible to the original methods (or, in the case of Kellow & Jones, 2008, improve on the original methods to remove confounds). Stoet and Geary (2012) submitted their article in late 2011, which is exactly when the replication crisis in psychology was first starting. Psychologists now recognize that studies that the best replications are those that resemble — as closely as possible — the original study, with the same materials, procedures, sample characteristics, etc.

But there is room for disagreement about what makes a study a “real” replication. Even if one accepts Stoet and Geary’s (2012) broader definition of a replication, the finding that only 30% of studies can find stereotype threat in females does not inspire confidence in the existence of the phenomenon. It is, however, in line with findings that social psychology research replicates about one-quarter of the time (Open Science Collaboration, 2015).

References

Finnigan, K. M., & Corker, K. S. (2016). Do performance avoidance goals moderate the effect of different types of stereotype threat on women’s math performance? Journal of Research in Personality, 63, 36-43. https://doi.org/10.1016/j.jrp.2016.05.009

Gibson, C. E., Losee, J., & Vitiello, C. (2014). A replication attempt of stereotype susceptibility (Shih, Pittinsky, & Ambady, 1999). Social Psychology, 45(3), 194-198. https://doi.org/10.1027/1864-9335/a000184

Kellow, J. T., & Jones, B. D. (2005). Stereotype threat in African-American high school students: An initial investigation. Current Issues in Education, 8(20), 1-11.

Kellow, J. T., & Jones, B. D. (2008). The effects of stereotypes on the achievement gap: Reexamining the academic performance of African American high school students. Journal of Black Psychology, 34(1), 94-120. https://doi.org/10.1177/0095798407310537

Moon, A., & Roeder, S. S. (2014). A secondary replication attempt of stereotype susceptibility (Shih, Pittinsky, & Ambady, 1999). Social Psychology, 45(3), 199-201. https://doi.org/10.1027/1864-9335/a000193

Collaboration, O. S. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

Shih, M., Pittinsky, T. L., & Ambady, N. (1999). Stereotype susceptibility: Identity salience and shifts in quantitative performance. Psychological Science, 10(1), 80-83. https://doi.org/10.1111/1467-9280.00111

Stoet, G., & Geary, D. C. (2012). Can stereotype threat explain the gender gap in mathematics performance and achievement? Review of General Psychology, 16(1), 93-102. https://doi.org/10.1037/a0026617

Warne, R. T. (2020). In the know: Debunking 35 myths about human intelligence. Cambridge University Press. https://doi.org/10.1017/9781108593298