Self-selection biases in psychological studies: Personality and affective disorders are prevalent among participants

Kaźmierczak I, Zajenkowska A, Rogoza R, Jonason PK, Ścigała D (2023) Self-selection biases in psychological studies: Personality and affective disorders are prevalent among participants. PLoS ONE 18(3): e0281046, Mar 8 2023. https://doi.org/10.1371/journal.pone.0281046

Abstract: Respondents select the type of psychological studies that they want to participate in consistence with their needs and individual characteristics, which creates an unintentional self-selection bias. The question remains whether participants attracted by psychological studies may have more psychological dysfunctions related to personality and affective disorders compared to the general population. We investigated (N = 947; 62% women) whether the type of the invitation (to talk about recent critical or regular life events) or the source of the data (either face-to-face or online) attracts people with different psychopathology. Most importantly, participants who alone applied to take part in paid psychological studies had more symptoms of personality disorders than those who had never before applied to take part in psychological studies. The current results strongly translate into a recommendation for either the modification of recruitment strategies or much greater caution when generalizing results for this methodological reason.

Discussion

The main aim of this project was to investigate self-selection biases related to the prevalence of personality disorders in psychological studies. We tested whether different types of research invitations attract different research participants in terms of their psychopathology. Indeed, people who replied to an advertisement on a study on a negative critical life event and its psychological consequences that took place up to two months before the research and led them to low mood had not only more personality disorders (PDs), but also the number of symptoms for different types of PDs compared to those who volunteered for a study on a regular life event and non-volunteers. Also, participants who replied to an advertisement on a study on a recent negative critical life event without the low mood requirement had more symptoms than those who volunteered for a study on a regular life event and non-volunteers. Still those who never participated in research before (i.e. non-volunteers) were likely to show the least symptoms of PDs compared to those who did, suggesting that people with the healthiest structure of personality (and reflecting the general population) are not usually included in research samples or are relatively rarely.

At the same time, personality disorders (more numerous and higher in volunteers) are associated with rigid (and maladaptive) beliefs and the resulting inflexible behavioral patterns [34, 35]), which may be of great importance in experimental research, particularly while interpreting the effectiveness of an experimental manipulation, but also in the identification and/or description of any psychological phenomena. Many studies show that participants with PDs demonstrate specific attentional coping styles [36] and biased attention to emotions and facial expressions [37, 38], which might interplay with all experimental procedures.

Comparing the studied groups to the general population, it should be noted that both the ratio of participants who met the criteria of a PD (from six to 75% depending on a type of PDs and an advertised study) and the number of clinically diagnosed PDs (from three to six coexisting PDs dependent on a type of an advertised study) are unexpected outcomes. All comparison groups differed to a greater or lesser extent from the distribution of PDs in the general population, regardless the wide range of the results. Studies show that it ranges from 4.4% to 13.4% [14, 39] for the European population and from 9.0% to 21.5% [3, 40] for the United States population. The highest overall prevalence of PDs (equal to 45.5%) has been identified amongst psychiatric patients [31], and it is still lower than in one of our advertised study (i.e., after a critical life event that took place up to two months before the research and led to low mood).

In addition, the prediction that the personality organization of both online participants and volunteers who applied for different types of face-to-face studies will be more pathological in comparison with non-volunteers was verified in the project. However, there were no differences in the intensity of Borderline PD symptoms, nonetheless volunteers who participated in a critical life event study and with low mood were diagnosed with this disorder most frequently. Although such key words included in the research invitation as “low mood” and “negative critical life event” had the power to attract people with particularly increased personality psychopathologies, it should be noted that all volunteers, regardless the experimental group, were characterized with its higher level. It might suggest that apart from participation in a study, they might indirectly seek for a psychological help and for a reason, however this hypothesis requires further investigation.

Furthermore, online participants were higher on depression and anxiety (their mean scores indicate clinical “caseness” using the cut-off of 11 points suggested by Zigmond and Snaith [31] as compared to those who never participated in research before (their mean scores indicate a borderline level for depression and clinical “caseness” for anxiety; [31] This finding is in line with some previous studies [13], however it is important to acknowledge that the online survey was conducted during COVID-19, which might have an aversive impact on participants. As Bueno-Notivol et al. [41] showed in their meta-analysis, the prevalence of depression (25%) in COVID-19 is even seven times higher compared to its global estimation in 2017. At the same time, although it causes a methodological restriction to adequately compare all groups, we cannot ignore the fact that all data collected during the pandemic is based on this specific research samples (e.g., most research is done remotely). Hence, paradoxically, the Internet sample from our study delivers a characteristic of the subjects who we are actually being studied now.

Additionally, our study was conducted in the early stages of the COVID-19 pandemic. This means that the heightened state of anxiety that emerged in almost everyone was not able to alter enduring dispositional personality traits. As longitudinal studies have shown, change is a dynamic and temporal process. A longitudinal study of Caldioroli et al. [42] involving 166 individuals affected by different psychiatric disorders at three time points (t0 as pandemic outbreak, t1 as lockout period, t2 as re-opening) showed significant deterioration during the lockout period with little improvement during the re-opening. Moreover, only psychopathology in patients with schizophrenia and obsessive-compulsive symptoms were not significantly improved at t2. Individuals with PDs were at higher risk for overall psychopathology than those with depression and anxiety/obsessive-compulsive and exhibited more severe anxiety symptoms than schizophrenic patients.

Summing up, as (1) volunteers vary in terms of psychopathology depending on the type of both invitation and study they wanted to participate in, and also differ from those individuals who usually do not come to psychological research, (2) there is no basis for assuming that the presented findings are an isolated case. Hence, it is advisable to interpret all psychological research outcomes considering the impact of the form of invitation to the research and the type of research itself on the potential psychopathology of the participants. Moreover, the research outcomes need to be interpreted in close connection with the finding of larger psychopathology of the volunteers compared to non-volunteers. This conclusion translates into a recommendation for either the modification of recruitment strategies or much greater caution when generalizing results for this methodological reason.