Overall, Mitchell found that lab findings usually replicate in the real world (r = .71, where 1 would be a perfect match), but the devil is in the detail: some sub-disciplines in psychology fared much better than others; the size of the effects often differed greatly between lab and real world; and in a worrying number of cases, the real world results were actually in the opposite direction to the lab findings.
"Many small effects from the laboratory will turn out to be unreliable," Mitchell concluded, "and a surprising number of laboratory findings may turn out to be affirmatively misleading about the nature of relations among variables outside the laboratory."
Breaking the results down by sub-discipline, findings replicated from the lab most often in Industrial-Organisational Psychology (based on 72 comparisons) and least often in Developmental Psychology, where the three comparisons showed the average field result was actually in the opposite direction to the lab findings. The massive discrepancy in number of comparisons in these sub-disciplines makes it difficult and unfair to draw any definitive conclusions from this particular contrast. However, Social psychology had a similar number of comparisons (80) to Industrial Organisational Psych, yet produced a far lower replication rate (r = .53 vs. r = .89). Mitchell said further research is needed to find out why this might be.
There were also important differences in replication rates (from lab to field study) within different psychology sub-disciplines. For example, Industrial Organisational Psychology studies of performance evaluations translated less well from the lab compared with other topics of study in that discipline. Across subfields, lab studies of gender differences were particularly unlikely to translate to the real world. "We should recognise those domains of research that produce externally valid research," Mitchell said, "and we should learn from those domains to improve the generalisability of laboratory research in other domains."
Mitchell, G. (2012). Revisiting Truth or Triviality: The External Validity of Research in the Psychological Laboratory. Perspectives on Psychological Science, 7 (2), 109-117 DOI: 10.1177/1745691611432343
Further reading: Gregory Mitchell contributed to The Psychologist's current opinion special on replication in psychology (free access).
Post written by Christian Jarrett for the BPS Research Digest.
You have read this article Methodological with the title Do psychology findings replicate outside the lab?. You can bookmark this page URL http://psychiatryfun.blogspot.com/2012/04/do-psychology-findings-replicate.html. Thanks!