Beyond questionable research methods: The role of omitted relevant research in the credibility of research

Schmidt, Frank L. (2017). Beyond questionable research methods: The role of omitted relevant research in the credibility of research. Archives of Scientific Psychology, 5(1), 32-41. http://dx.doi.org/10.1037/arc0000033

ABSTRACT: Governments often base social intervention programs on studies done by psychologists and other social scientists. Often these studies fail to mention other research suggesting that such interventions may have a limited chance of actually working. The omitted research that is not mentioned often shows that the behaviors and performances targeted for improvement by the environmental intervention programs are mostly caused by genetic differences between people and for that reason may be more difficult to change than implied in these studies. This is particularly true when the goal is to greatly reduce or eliminate differences between people in such domains as school achievement, impulsive behaviors, or intelligence. This problem of omitted research creates two problems. It tends to call into question the credibility of all social science research, even the studies that do not omit relevant research. And from an applied point of view, it leads to the expenditure of taxpayer dollars on programs that are unlikely to produce the desired outcomes.

SCIENTIFIC ABSTRACT: This article explores an important credibility problem in the research literature beyond the issue of questionable data analysis methods: the problem of omission of relevant previous research in published research articles. This article focuses on this problem in 2 areas: (a) studies purporting to demonstrate the effects of people’s experiences on their later life outcomes while failing to discuss or mention the probable causal role of genetic inheritance in producing these effects, despite the strong evidence for this connection from behavior genetics research; and (b) studies of specific aptitudes (specific abilities) such as verbal, spatial, or reasoning that fail to acknowledge or mention that such aptitudes are indicator variables for general mental ability (GMA; or intelligence) and that after proper control for GMA the residuals in these aptitudes make essentially no contribution to prediction of real world academic, occupational, or job performance. It is only the GMA component in such aptitudes that produces the ability to predict. As is well known today, the issue of the credibility of research conclusions is prominent (Ioannidis, 2005). In both the areas examined in this article, these deficiencies create serious and unnecessary credibility problems, and the doubts they inspire about credibility could unfortunately be generalized to other research areas in which these problems do not exist.

KEYWORDS: research credibility, behavior genetics, general mental ability, intelligence, specific abilities

---
The purpose of this article is to draw attention to an important problem in the research literature beyond the issue of questionable statistical data analysis methods in research. That is the problem of the omission of relevant research findings in published research articles. This article has focused on this problem in two areas. The first area consists of studies purporting to demonstrate the effects of people’s experiences on their life outcomes. Many studies drawing causal conclusions about these experiences fail to mention the probable causal role of genetic inheritance in producing these effects, despite the overwhelming evidence for this connection from behavior genetics research. The second area consists of studies of specific aptitudes (specific abilities) such as verbal, quantitative, and spatial, or reasoning. Many such studies fail to acknowledge or even mention that such aptitudes are indicator variables for GMA and that after proper control for GMA the residuals in these aptitudes make essentially no contribution to prediction of real world academic, occupational, or job performance. It is only the GMA component in such aptitudes that produces the ability to predict.

A reviewer suggested that research areas beyond the two examined in this article should be addressed. However, the first area examined here is very broad and includes research in many areas and specialties of psychology and the social sciences. These include the research areas of academic achievement, aggressive behavior, abusive behavior within families, failure in intimate relationships, authoritative versus authoritarian parenting, effects of parental reading on children, later effects of self-control displayed in childhood, and effects of personality traits on behaviors and perceptions. These multiple areas of research are all observational or correlational. Areas of experimental research are not included because (in theory) randomization of subject assignment controls for hidden or unrecognized causal variables (lurking variables; Joiner, 1981). However, experimental psychology research as typically found in the literature, although not subject to the problems examined in this article, has other serious problems; for example, see Schmidt and Oh (2016). In addition, Jussim, Crawford, Anglin, Stevens, and Duarte (2016) discussed the tendency in experimental social psychology for researchers to fail to consider alternative and plausible interpretations of their findings and pointed out that this omission reduces the credibility of their research conclusions.

The issue of the credibility of research conclusions is prominent today (Ioannidis, 2005). In both the areas examined in this article, these deficiencies create serious and unnecessary credibility problems, and the doubts they inspire about research credibility could unfortunately be generalized inappropriately to other research areas in which these problems do not exist. So it is important that this problem be addressed and corrected.

However, it is also important not to leave the impression that all studies have these problems. Studies can be found that are exemplary. One example is the study by Dinescu et al. (2016). This study examined the hypothesis that when people marry, their level of alcohol consumption decreases. They acknowledged that marriage per se might not be the causal variable and that people who marry might be genetically different from those who do not. So they controlled for genetic effects using a sample of 1,703 monozygotic and 722 dizygotic twins. Their results showed the even after controlling for genetic effects, people drank less after marriage than they had before marriage. Another exemplary study is Gotlib et al. (2015). This study found that apparently healthy women whose mothers suffered from depression had shorter telomeres and greater cortisol reactivity to stress than did women whose mothers had never been depressed. They acknowledged that this effect could be genetic or environmental or both, and they called for research to determine the relative contributions of genetic and environmental causes. There is another particularly important example. For over 25 years, Terrie Moffitt and Avshalom Caspi have conducted longitudinal research studies that have taken into account both genetic and environmental contributions to  human behaviors. And in 2016 they received the Distinguished Scientific Contributions Award for this work from the American Psychological Association (Award for Distinguished Scientific Contributions, 2016). These and other examples in the literature show that it is possible to conduct research of the type called for in this article.

The hope is that the information presented in this article will lead to recognition in the literature of the role of behavior genetics findings in studies interpreting relationships between experiences and later life outcomes; and to recognition of the central role of GMA in studies examining specific aptitudes and abilities. At present the literatures in these two areas contain many studies that are scientifically incomplete. These changes are important for establishment of the credibility of research conclusions in these areas and may help to deter credibility losses across other areas of research.