PhD Thesis from 2018: The Human Fallibility of Scientists - Dealing with Error and Bias in Academic Research

Veldkamp, Coosje L. S. 2017. “Doctoral Thesis: The Human Fallibility of Scientists - Dealing with Error and Bias in Academic Research.” PsyArXiv. October 19. doi:10.31234/osf.io/g8cjq

Abstract
Recent studies have highlighted that not all published findings in the scientific literature are trustworthy, suggesting that currently implemented control mechanisms such as high standards for the reporting of research methods and results, peer review, and replication, are not sufficient. In psychology in particular, solutions are sought to deal with poor reproducibility and replicability of research results. In this dissertation project I considered these problems from the perspective that the scien¬tific enterprise must better recognize the human fallibility of scientists, and I examined potential solutions aimed at dealing with human error and bias in psychological science.

First, I studied whether the human fallibility of scientists is actually recognized (Chapter 2). I examined the degree to which scientists and lay people believe in the storybook image of the scientist: the image that scientists are more objective, rational, open-minded, intelligent, honest and communal than other human beings. The results suggested that belief in this storybook image is strong, particularly among scientists themselves. In addition, I found indications that scientists believe that scientists like themselves fit the storybook image better than other scientists. I consider scientist’s lack of acknowledgement of their own fallibility problematic, because I believe that critical self-reflection is the first line of defense against potential human error aggravated by confirmation bias, hindsight bias, motivated reasoning, and other human cognitive biases that could affect any professional in their work.

Then I zoomed in on psychological science and focused on human error in the use of null the most widely used statistical framework in psychology: hypothesis significance testing (NHST). In Chapters 3 and 4, I examined the prevalence of errors in the reporting of statistical results in published articles, and evaluated a potential best practice to reduce such errors: the so called ‘co-pilot model of statistical analysis’. This model entails a simple code of conduct prescribing that statistical analyses are always conducted independently by at least two persons (typically co-authors). Using statcheck, a software package that is able to quickly retrieve and check statistical results in large sets of published articles, I replicated the alarmingly high error rates found in earlier studies. Although I did not find support for the effectiveness of the co-pilot model in reducing these errors, I proposed several ways to deal with human error in (psychological) research and suggested how the effectiveness of the proposed practices might be studied in future research.

Finally, I turned to the risk of bias in psychological science. Psychological data can often be analyzed in many different ways. The often arbitrary choices that researchers face in analyzing their data are called researcher degrees of freedom. Researchers might be tempted to use these researcher degrees of freedom in an opportunistic manner in their pursuit of statistical significance (often called p-hacking). This is problematic because it renders research results unreliable. In Chapter 5 I presented a list of researcher degrees of freedom in psychological studies, focusing on the use of NHST. This list can be used to assess the potential for bias in psychological studies, it can be used in research methods education, and it can be used to examine the effectiveness of a potential solution to restrict oppor¬tunistic use of RDFs: study pre-registration.

Pre-registration requires researchers to stipulate in advance the research hypothesis, data collection plan, data analyses, and what will be reported in the paper. Different forms of pre-registration are currently emerging in psychology, mainly varying in terms of the level of detail with respect to the research plan they require researchers to provide. In Chapter 6, I assessed the extent to which current pre-registrations restricted opportunistic use of the researcher degrees of freedom on the list presented in Chapter 5. We found that most pre-registrations were not sufficiently restrictive, but that those that were written following better guidelines and requirements restricted opportunistic use of researcher degrees of freedom considerably better than basic pre-registrations that were written following a limited set of guidelines and requirements. We concluded that better instructions, specific questions, and stricter requirements are necessary in order for pre-registrations to do what they are supposed to do: to protect researchers from their own biases.

Letters To A Spanish Youngster II

Letters To A Spanish Youngster II
[...]

Su Señoría reina de los jardines*, Who sets the city on fire**, while causing a little !  in Her subjects' hearts,

At first I thought to dispense with naming the many beautiful things that compound un facetado diamantino tan bonito como el Suyo, my lady and master, since I have not the necessary talent to write lists of pretty things like those lists of, just to name one, Walt Whitman†:

         [Por las salinas, por los naranjales o las sombras de los pinares,
         [...]
         Complacido con la gente del país y con los extranjeros, contento con lo nuevo y lo viejo,
         Contento con la mujer envejecida y contento con la que es linda,
         [...]
         Caminando esa misma tarde con la cabeza alzada hacia las nubes por una callejuela o 
                                                                                                                        [por la playa,
         Paseando del brazo de dos amigos y yo en el medio,
         [...]
         Lejos del campamento, estudiando las huellas de los animales o de los mocasines,
         Junto a la cama en el hospital, sirviendo limonada al que tiene fiebre,]


(Dios mío, qué bonito es el último verso.)

Pero no puedo evitarlo, voy a enumerar igualmente.     "¿Me contradigo? Muy bien, me contradigo."†

                        :-))

y, muy deficientemente, nombro como elementos de la gran composición visible que es un ángel como Vd. los primeros que me vienen a la mente, en el orden en que se me aparecen (si bien no es que ninguno tenga más mérito que otro elemento por el hecho de aparecer antes):

         *  una   ҉  sonrisa  ҉   que hechiza
         *  Sus   ҉  ojos  ҉   luminosos y despiertos, ante cuyos rayos tengo que retirar la mirada, confundido
         *  una   ҉  forma tan elegante y cuidada de vestir  ҉  que me hace temer que los demás presentes me amonesten por no poder dejarla de contemplar
         *  Su suave   ҉  acento  ҉
         *  los   ҉  cabellos  ҉  , y color y forma de los mismos, arrebatadores, como sin duda Vd misma sabe
         *  la   ҉  garganta  ҉   de una Afrodita o Diana o Apollo de las que esculpieron aquellos grandes artistas
         *  el   ҉  conjunto de Su cuerpo  ҉  , unas formas sublimes que mueven a la piedad y el temor a Sus servidores
         *  lo que se adivina como preciosas   ҉  manos  ҉  , aunque desde la distancia no podía verlas
         *  la cálida nieve del   ҉  rostro  ҉   ‡
         *  una   ҉  gracia del movimiento  ҉   comparable (y al comparar, superior) a la de la Victoria que esculpieron tras Samotracia, o del David de Miguel Ángel y cuyas fotos incorporo para contraponerlas con Su Señoría:

                      [...]

, and above all,   ҉  a demeanor, and general attitude  ҉   so charming as to cause such a great effect as it is already falling upon me.

Milton spoke֍ of greatness (although unrelated to this I am talking about), which I think is applicable to Your Honor, Queen of Gardens:

         "[...] Not Babylon,
         Nor great Alcairo such magnificence
         Equall'd in all their glories"

, and that is what I feel seeing You with those great looks of You, and hearing You talking to others. Oh, that voice of Your Honor...

And then...

I also see Your smile and the other visible corporeal, physical attributes (those   ҉  eyes  ҉   of Yours, the free hair, Your face, everything I mentioned above), as impressive as the incorporeal ones, and I wonder at the mystery of Your mind working, interacting with we mere mortals, in awe hearing the waves of Your voice, one of the physical media with which Your spirit communicates with us.

---
I already know that it is impossible to get closer to Your Honor as much as I would, and it is true that that makes me a bit sad. Petrarca wrote that Cupid says, about the effect on those who are deeply moved by persons like You‡:

         [write down in golden letters what you've seen          [escribe en letras de oro lo que viste;
         how I change those who follow me in color                cómo a mis seguidores hago palidecer
         and in an instant make them live and die.]                 y en un mismo instante los vuelvo muertos y
                                                                                                                                                          vivos.]

OK, it is a bit of an exaggeration that part about the dead, isn't it. But that about the change in color and feeling bad when meeting Your Grace it is something that surely happens, at least to this follower of Yours. Despite the fear that I feel, I cannot agree with Love when he says‡ [yo me nutro de lágrimas]; on the contrary, those who adore You, my governor, must be happy for getting to know about You and met You and for this we are bound to devote to You beautiful thoughts, like this one‡:

        [Flowers joyful and glad, fortunate grass                      [Alegres y felices flores, afortunadas hierbas
        on which my lady used to walk in thought,                   que, pensativa, mi señora pisar suele;
        shore that would listen to her words of sweetness      prado que escuchas sus dulces palabras,
        conserving traces of her lovely foot,].                           y del bello pie algún vestigio guardas;].


We may always object that the visible part, Your beauty, dominates us... But a Persian mystic replied to this (or so they say J Rūmī wrote)¶:

[The body moves by means of the spirit, but you do not see the spirit: Know the spirit through the body's movement!]

Some day I will dare to ask Your Honor for a hearing in which to make some offerings, madonna, like a beautiful book of the arts (there are true jewels, things so impressive!), or a ticket for some museum, exhibition, movie, ballet or anything You may like. Until now, I didn't gather enough strength   :-(     , since there is the risk that this could happen to me‡‡:

[...]

, deservedly, of course.

With envy of the little insect that overpasses Your Honor while reading this letter, and great interest in Your health and that of Your friends and family, and wishing that Your days are devoid of sorrow, Your

                 devout admirer

--
Notes

*  Cantar de los Cantares, 8.13, Biblia de Jerusalén. Barcelona: Desclée De Brouwer, 2009.

** Anonymous epigram, in 'The Greek Anthology' translated by W R Paton, 1916, book V, 2, 1.

†  Adapted from W Whitman's 'Leaves of Grass' and José Martí & Jorge Luis Borges's translations.

‡  Adapted from Francesco Petrarca, Petrarch Songs and Sonnets, A Bilingual Selection, translated by Richard Kilmer (London: Anvil Press Poetry, 2011), Petrarch: The Canzoniere, or Rerum Vulgarium Fragmenta, translated by Mark Musa (Bloomington: Indiana University Press, 1996), and Atilio Pentimalli's translation (Barcelona: Ediciones Orbis, 1998): CLVII, 9; XCIII, 2-4; XCIII, 14; CLXII, 1-4.

֍  John Milton's Paradise Lost, 1674 edition, i. 717-9

¶  Jalāl Rūmī's Masnavi IV, 155, apud William C Chittick's The Sufi Path of Love: The Spiritual Teachings of Rumi. Albany, NY: State University of New York Press, 1983. Page 28.

‡‡  [...]

[...]

Letters To A Spanish Youngster I

Letters To A Spanish Youngster I
[...]

Your Honour the little good wizard-magician / Meritíssima, a pequena maga-bruxa boa / Su Señoría maga y brujita buena,

I finally dare, for the reason I explain later▼, to write to Your Grace after many months since the moment in which I had the privilege of breathing for the first time in the same room that You were in, blessed person.

(Although [tanto tengo para decirle que no me atrevo a empezar.]*    :-)    )

This cannot be done in any other way than speaking to You in an extremely respectful manner, so that it is made explicit the enormous distance between such a special person as Your Honor and Your devout admirer, who waits for the day to contemplate You in a so elevated position from the pedestal of that altar** in which You are*:

        "I' benedico il loco e 'l tempo et l'ora                    [I bless the place, the time and hour of the day
        che sì alto miraron gli occhi mei,                          that my eyes aimed their sights at such a height,
        et dico: Anima, assai ringraziar dei                      and say: 'My soul, you must be very grateful
        che fosti a tanto onor degnata allora."                 that you were found worthy of such great honour.]


Maybe I misspoke saying "admirer," mi persona especial. A passionate poet, if any other class is possible, is, actually, ADORADOR. In imitation of the true poets, even knowing I have no talent to be of their numbers, I consider myself an idolater of Your complex person – the combination of the visible person and the essence, we could say spiritual essence, that we envision there is beyond what is visible (la intuición me la ha dado verla sonreír cuando habla con compañeros Suyos).

These prodigies of Your character, hidden from us at first because they were not immediately apparent, comprise the enigmatic and mysterious component of a Grace we all were waiting for.

Those unfortunate artists of past centuries didn't paint or carve a fourth Grace because Your sacred steps had not marked their imprint on this sad planet and there was no certainty, until recently, that there were more than three ones.

Pero nosotros, Sus súbditos de hoy, no tenemos dudas: somos más afortunados, hemos coincidido en el tiempo con Su presencia y sabemos que el cuarto regalo que nos hacen los dioses a los hombres es Su Señoría, de nombre ignoto.

---
▼
It is difficult to take the decision to write down some words, and much harder to deliver them. Many good poets and musicians regret to leave extant their poor works, always defective, insufficient ... and I'm just a troubadour. Although these texts are not worthy of Your Honor's attention, I finally found the courage and audacity to communicate with You.

Le hago llegar estas líneas con motivo del Primer Día Internacional de Glorificación de la Persona Mágica, que se celebra hoy y que acabo de instituir ayer mismo tras pensar en Vd.

(No se preocupe Su Señoría, no es que deje de pensar en Su persona por muchos minutos... Simplemente se me ocurrió ayer tras una de las veces en que me acordé de lo agradable que resulta ver y escuchar Su parte física cuando tengo la oportunidad.)

On the occasion of this solemn day I dare to steal some of Your time to read some thoughts† that I believe are very beautiful and appropriate for this day, even risking that they are just funny to You:

[Woman is certainly within her rights, and she even performs a kind of duty when she endeavors to appear magical and supernatural; she should dazzle men and charm them; she is an idol who should be covered with gold in order to be worshipped.

She should therefore borrow from all the arts the means of arising above nature in order better to subjugate all hearts and impress all minds. 

It is of no consequence that her ruse and artifice be known by all, if their success is certain and their effects always irresistible.]

(Of course I do not necessarily agree with all the texts I print here, nor do I endorse the authors' inclinations, politics, life, etc.)

---
I am fully conscious that if the writings that I make mention of in this and following (!) letters are of Your liking, it is applicable to me what Sir John Suckling said‡:

        "But the spite on ’t is, no praise                              [Pero a pesar de esto, ningún encomio
        Is due at all to me:";                                                me es debido];

is Your Grace, mi señora mística, the one that inspire to uncover beautiful things to be able to send them to You.

Yo quiero que pueda Vd. disfrutar de lo que es bello y agradable. Por eso, además de lo que otros escriben de forma admirable, adjunto una imagen y su calco. No tengo dote ninguna para pintar, así que, para Vd., reduzco a líneas básicas cosas hermosas que veo y se las muestro. Sé que es de muy poco valor, pero en ausencia de capacidad artística, pongo mi esfuerzo, aunque sea una pobre compensación     :-(    .

I've got several letters like this in preparation. May Your Honor authorize me to share them with You? You may authorize me writing to my e-mail֍. There is no need for me to see Your name, mia senhor, I may send them to the name of a good friend of Yours in the same office and then she will pass the letters to You. This way, Your identity will be safe. Simplemente recibe cartas bonitas y sonríe mientras las lee     :-)        .

Of course it is advisable, for Your peace of mind, to write me from an e-mail not connected to You.

[...]

Soy consciente de que no debería decir a nadie que le estoy escribiendo, mi señora, pero no puedo hacerle llegar estas notas si no lo comento al menos una vez. Tomando palabras de una poetisa rusa¶,

        [Ya me es indiferente en qué lenguaje no seré comprendida por el hombre.].


Seguiré escribiéndole si Su compasión me lo permite. ¿Sería correcto una carta por semana? Intentaré que no sean más largas que esta.

With deep respect and a kiss in one of the prints, already erased, that Your delicate soles left the day before on the sand of a beach imagined by the poets (con profundo respeto y un beso en  una  de las huellas, ya borradas, que en la arena de una playa imaginada por los poetas dejaron el día anterior Sus delicadas plantas), se despide, sinceramente Suyo,

                 admirador rendido‡‡  ante Su Señoría


‡‡ (I cannot think of printing my name in the same paper than Your Honor's unless it is done in non-capital letters.)

--
Notes

*  Adapted from Petrarch Songs and Sonnets, A Bilingual Selection, translated by Richard Kilmer (London: Anvil Press Poetry, 2011), Petrarch: The Canzoniere, or Rerum Vulgarium Fragmenta, translated by Mark Musa (Bloomington: Indiana University Press, 1996), and Atilio Pentimalli's translation (Barcelona: Ediciones Orbis, 1998),  CLXIX, 14; XIII, 5-8.

Spanish version:

[Y bendigo el lugar y el tiempo y la hora
en que mis ojos tan alto miraron,
y digo: Alma, mucho has de agradecer,
que de tanto honor entonces fuiste honrada.]

I take all the blame for any translation faults.

This is valid for any of my writings. "Traduttore, traditore".

**  Plato's Phaedrus, 254b. Available at http://classics.mit.edu/Plato/phaedrus.html:

"And now they are at the spot and behold the flashing beauty of the beloved; which when the charioteer sees, his memory is carried to the true beauty, whom he beholds in company with Modesty like an image placed upon a holy pedestal."

Bad translation, it substitutes a girl for a boy ("the beloved" of the "flashing beauty")...        :-)        .

†  Charles Baudelaire translated by Wallace Fowlie. New York: Bantam Books, 1964. Eloge du maquillage.

La femme est bien dans son droit, et même elle accomplit une espèce de devoir en s’appliquant à paraître magique et surnaturelle ; il faut qu’elle étonne, qu’elle charme ; idole, elle doit se dorer pour être adorée. Elle doit donc emprunter à tous les arts les moyens de s’élever au-dessus de la nature pour mieux subjuguer les cœurs et frapper les esprits. Il importe fort peu que la ruse et l’artifice soient connus de tous, si le succès en est certain et l’effet toujours irrésistible.

‡  Sir John Suckling's 'The Constant Lover', ca. 1630. Available at http://www.bartleby.com/40/234.html.

֍  [...]

¶  Marina Tsvetaeva's Antología poética. Madrid: Hiperión, 2014. Translated by Lola Díaz, version by Severo Sarduy, p. 163.