Gender and evaluation criteria for tenure and promotion: At the top quality level, there are no differences between males and females in the probabilities of preferring bibliometric criteria

What should be rewarded? Gender and evaluation criteria for tenure and promotion. Laura Cruz-Castro, Luis Sanz-Menendez. Journal of Informetrics, Volume 15, Issue 3, August 2021, 101196. https://doi.org/10.1016/j.joi.2021.101196

Highlights

• Both interest-based and cognitive factors account for the preference of bibliometrics as evaluation criteria.

• Individual research quality influences positively the preference for bibliometrics.

• Differences in probabilities across genders are significant but small.

• At the top quality level, there are no differences between males and females in the probabilities of preferring bibliometric criteria

Abstract: Criteria for assessing candidates are essential elements for the functioning of evaluation practices in academia. This article addresses a relevant issue of academia: the preference for evaluation criteria for tenure and promotion, as reported by female and male academics employed at Spanish universities. We use survey data from 4,460 faculty members, testing whether there are differences in the evaluation criteria that women and men prefer and exploring the factors that account for such preferences. Our focus is on bibliometric evaluation criteria. We propose an analytical model that considers the influence of career and quality factors, values about universalism and the mission of universities, and beliefs about meritocracy in the context of the academic evaluation system. We use a binary logistic model to explain the preference for bibliometric criteria and develop the comparisons by gender using predicted probabilities and marginal effects for estimating the difference. We find that female academics do not have the same preferences as men and report lower preferences for bibliometrics. However, women at the highest research quality levels have similar probabilities than males to prefer bibliometric criteria for evaluation.

Keywords: Bibliometric assessmentResearch evaluationMerit criteriaTenure and promotionValue JudgmentGender disparitiesAcademic values and beliefsPreferencesBibliometric evaluation criteria

5. Discussion

Our empirical findings do not support the existence of a dominant preferred academic evaluation criterion for tenure and promotion. In the views of Spanish academics, it does not appear that any single criterion suffices for getting tenure and promotion. Reported preferences for evaluation criteria fit only imperfectly with the norms of scientific meritocracy and the Mertonian rule, but this does not mean that they are random or meaningless.

As indicated by Musselin (2013), two points lead to some relativization of the universalistic scientific reward principle: first, selection processes are rarely driven exclusively by the research criterion; candidates’ teaching experience is systematically taken into account; second, since candidates’ scientific production is not likely to be directly evaluated, peers prefer to give credence to standardized indicators that allow them to assess scientific value indirectly, especially the number of publications and citations.

Our general model to address the preferences is consistent, but other explanatory variables should be incorporated17. Analytically, both interest-based and cognitive variables are needed to account for preferences and their relevance is supported by the empirical data.

It is interesting that academics who have developed their career in a single university (inbred) do not show significantly different preferences regarding bibliometric criteria than their more mobile counterparts. Among the career factors, full professorship and PhD abroad strongly predict the preference for bibliometric criteria. This relationship may reflect the circumstance that full professors, once having obtained that rank (Ridgeway, 2014) , will no longer take part in competitions for jobs (except in the unlikely event of moving to a different university), and thus they do not view their potential colleagues as future competitors, thereby reducing the risk of adverse selection (Carmichael, 1988). This ‘competition’ explanation is consistent with previous research reporting lower levels of conflict in decisions regarding tenure for departments with a larger number of full professors (Hern and Anderson 2002).

Overall, the results reveal that individual research quality, as a variable that could be related with self-interest, when controlling for other covariates, does play a relevant role in accounting for the preference of bibliometric criteria in evaluation; those with higher research quality are more in favor of bibliometric criteria, a finding consistent with recent research (Langfelt, Reymert and Aksnes 2020; Reymert, Jungblut, & Borlaug, 2021).

Academics integrate concerns about the university and the operation of academic evaluation into their preferences. The idea that teaching and training is the principal university mission stands out as one of the strongest predictors for not including bibliometric criteria among the evaluation preferences. In general, those who believe that the level of demand for merit requirements in place at the national accreditation agency are poor tend to prefer bibliometrics. Such a relationship may indicate the belief that the use of bibliometric criteria could be a way of palliating some of these shortcomings. Additionally, the belief that the current accreditation system is the best to assure a merit-based selection influences the preference for bibliometric criteria positively, possibly because standardized metrics increase trust in accreditation procedures (Sanz-Menéndez & Cruz-Castro 2019).

At this point we have to acknowledge that the relationship to values may operate both ways, and that having a preference for bibliometric criteria may influence the negative appraisal of a system that has been criticized in the past as suffering from cronyism (Bosch, 2001).

Central to our research questions, the analysis has revealed that female academics do not have the same preferences for evaluation criteria as men, and tend to prefer bibliometric less than their male counterparts; the marginal effect of gender is 7.8%, but for women at the highest level of quality the probabilities of preferring bibliometrics are almost the same than for males. This finding is consistent with recent evidence from Langfelt, Reymert and Aksnes (2020) showing that the use of bibliometrics in peer evaluation is higher among reviewers with higher scores on these metrics.

There are some plausible lines of interpretation of the differences. It might be the case, as found by (van den Brink and Benschop, 2012), that despite the gender neutrality of academic evaluation criteria, women at low to medium levels of quality perceive they would be disadvantaged if evaluated by certain criteria, if opportunities to develop and perform along them are unequally distributed ex ante. The literature on publications and productivity and impact differences by gender is substantial. Classical sociology of science and recent bibliometric research have extensively addressed the issue. The general empirical claim has been that men on average publish more papers and receive more citations than female scientists (Thelwall, 2018) . This could be a potential source of differences in attitudes or preference towards the use of bibliometric criteria in tenure and promotion evaluation. A trend towards closing the gap has also been reported in the literature (Xie & Shauman 2003) as well as the finding that productivity of both men and women increases with scientific rank (Long et al. 1993). Regarding impact, according to Lariviere and Sugimoto (2017) the average impact factor of journals for men and women are much closer to parity than citations and, systematically, the gap in citations is much greater than the gap in impact factors, and always in favor of men. In other words, even when women publish their work in high impact factor journals, this work is cited less (Abramo et al. 2015), both in absolute and relative terms, but there is recent evidence that contradicts these findings (Thelwall 2018; Nielsen 2017).

It could also be the case, as reported by Niederle & Vesterlund (2011) in experimental studies, that women display different attitudes and responses than men toward competition even after controlling for performance, as they also found that men tend to be overconfident they will win in competitive environments, and women have a stronger tendency not to enter competitions. Much of the literature reports contradictory findings about females shying away from competition, and one of the reasons for this lack of consensus is that studies seldom consider feedback effects (Heilman et al. 2019). As some authors have highlighted, some academic evaluation practices and criteria (bibliometrics in our case) may be unappealing to women because of past histories of committees not hiring or not promoting women into high ranks, which lead to lower expectations of success (Fernández-Mateo and Kaplan 2018).

Two interesting empirical findings of our research are first, that both men and women at higher levels of quality prefer bibliometric evaluation criteria more than their colleagues at lower levels of research quality and, second, that the marginal effect of gender is almost non existing at higher levels of quality, meaning that there are no differences in probabilities of bibliometric preference for males and females at the top level of quality18.

A potential explanation for the first result could be connected with the efforts required by women to arrive at the highest level of quality recognition. Women at the top of research quality in our study have developed their careers in the 1990s or earlier, when there was a tight labor market, higher levels of competition and increased levels of evaluation requirements (Cruz-Castro and Sanz-Menéndez 2010); this generation could have interiorized the situation and give bibliometric criteria a more relevant role in evaluation (Sanz-Menéndez 1995). A different line of interpretation could be that women at the top assume the values and preferences of the “male dominated science system” (Derks et al. 2016).

Although differences by rank, values and beliefs matter, according to our data they follow similar patterns for women and men. These findings are also relevant for the analytical model. As mentioned, we need to account for structural, career, and cognitive variables to understand preferences for evaluation criteria in general, but when it comes to understanding the differences between female and male academics, probably the most important factors of those included in the explotatory model are structural and related to gender segmentation across quality levels and fields. However, females in different fields of science are more homogeneous, in comparison with males, in the probabilities to prefer bibliometric indicators.

We should acknowledge some caveats and limitations of the present study and the survey we have used (Sanz-Menéndez & Cruz-Castro (2019) provides more information). Although Spanish universities operate under the same national framework, they are under the political authority of their regional governments and these have diverse policies for universities and science, affecting the orientation and strategies of universities within their regions. We have addressed our research topic with a national survey (more ambitious than previous organization-focused case studies) and have not considered, this time, regional diversity; we leave this for future research, when factors related to the singularities of different universities could also be incorporated.

The definition of indicators used in the model to measure the three sets of explanatory variables could be further improved. Our analysis aims to be a contribution in the understanding of academic evaluation; further research could compare female and male preferences with the real operational processes of the tenure and promotion committees and with the explicit or formal criteria defined for this purpose.