A longitudinal investigation of the impact of psychotherapist training: Does training improve client outcomes? Seems not.

Erekson, D. M., Janis, R., Bailey, R. J., Cattani, K., & Pedersen, T. R. (2017). A longitudinal investigation of the impact of psychotherapist training: Does training improve client outcomes? Journal of Counseling Psychology, 64(5), 514-524. http://dx.doi.org/10.1037/cou0000252

Abstract: This study is a longitudinal examination of the impact of therapist stage of training on client outcomes in psychotherapy. The study included 22 PhD-level psychologists who work in a university counseling center (8 female, 14 male) who had completed at least 2 training periods in the center where data were gathered. Therapists worked with 4,047 clients, and 40,271 sessions were included in our analyses. Clients were given the Outcome Questionnaire-45 (OQ-45) on a session-by-session basis, tracking treatment response. The effect of stage of training on both the magnitude and speed of OQ-45 change was examined through hierarchical linear modeling. Therapists were found to achieve the same amount of change or less change on average in their later stages of training. Therapists were also found, on average, to achieve the same rate of change or a slower rate of change in later stages of training. Findings suggest that as therapists progress through formal stages of training, they do not improve in their ability to effect change in their clients. Given these findings, a better understanding of expertise in psychotherapy practice and how to develop it may be an important area for future theory development, research, and training program development. We call for further work examining if and how an individual therapist can become more effective with time.

Excerpts:

In the field of psychology, as in many professional disciplines, there is a belief among both the professionals and individuals served that training and experience (on the part of the practitioner) improve the quality of the service provided. In other words, as practitioners, we would like to believe that we not only influence outcomes, but that the outcomes of our clients improve as we gain training and experience. In support of this belief, doctoral level psychotherapist have highly structured training programs that they are required to complete in order to independently practice therapy. While it has been established that therapists differ in their effectiveness (Baldwin & Imel, 2013; Crits-Christoph et al., 1991; Kim, Wampold, & Bolt, 2006; Kraus, Castonguay, Boswell, Nordberg, & Hayes, 2011; Lutz, Leon, Martinovich, Lyons, & Stiles, 2007; Saxon & Barkham, 2012; Wampold & Brown, 2005), it is unclear to what extent training contributes to these differences, or to within therapist improvements.

The psychotherapy literature provides mixed results when examining the effect of stage of training and the related concept of therapist experience on psychotherapy outcome. Early studies by Bergin (1971) showed a positive relationship between therapist experience and patient outcome in 20 of 22 studies examined; other reviews, however, found no relationship or a negative relationship (r .01 in Smith & Glass, 1977; r .14 in Shapiro & Shapiro, 1982). Christensen and Jacobson (1994) concluded that the early evidence for the value of gaining professional experience is weak and suggested that training doctoral level psychotherapists is not justified. In the time since this suggestion, therapist experience studies have done little to refute this argument. Studies examining general therapist effects (e.g., Okiishi et al., 2006; Wampold & Brown, 2005) have found that some therapists have better outcomes than others, but that outcomes were not affected by the amount of experience the therapist had. A number of meta-analytic reviews have also been conducted, and have indicated a range of findings, from no effect to a modest effect for the relationship between experience and outcome (e.g., Berman & Norton, 1985; Crits-Christoph et al., 1991; Lyons & Woods, 1991; Stein & Lambert, 1995).

Level of training is perhaps one of the most intuitive definitions for therapist experience, and professional psychology is built on the premise that it is meaningful. Professional practice requires a structured graduate training program, where it is hoped that (contrary to Christensen and Jacobson’s (1994) assertion) the training experience will contribute to improvements in the therapist’s skill. Hill and Knox (2013) reviewed the evidence for changes in trainees’ helping skills and found limited and mixed evidence. They weighed results from analogue studies and self-reports by graduate students, and noted that the evidence for training effectiveness with “real clients” was limited by data being collected over short time periods, lack of control conditions, and a small number of cases (see pp. 782–784). They tentatively concluded that graduate training is effective, though cautioned that they could not rule out “confounds with other experiences” (p. 784).

Other studies examining the effect of training have addressed changes in therapist skills as well as clinical outcomes. Hill et al. (2015) studied multiple dimensions of change in trainees over the course of a period of training, finding evidence that trainees formed better treatment relationships, increased in their ability to facilitate improvements in client interpersonal relationships, and self-rated increases in ability to implement specific clinical skills and “higher order” therapist functioning. However, they did not detect changes in client engagement nor, most notably, in clients’ reductions in distress.

Researchers have also previously examined differences between specific stages of training. In a clinical benchmarking study in a university counseling center, Minami et al. (2009) found that interns and other trainees had pre- to posttreatment effect sizes that were significantly larger than those of staff clinicians, beyond what could be explained by differences in the number of sessions administered. Budge et al. (2013) further examined the effect of stage of training on outcomes and found that interns/postdocs achieved more change in psychological symptoms than licensed psychologists. Further, they found that interns/postdocs also achieved more change in life functioning than both practicum students and psychologists. Owen, Wampold, Kopta, Rousmaniere, and Miller (2016) found that trainees demonstrated improvements in outcomes over a 12-month period. However, client severity moderated this trend such that outcomes improved over time for less distressed clients but did not change for more distressed clients. In addition, they found no difference in the rate of therapist improvement by stage of training (with stages represented by cross-sectional data); that is, practicum students, interns, and postdoctoral therapists all improved with experience at the same rate.

Expanding the timeline to study the effect of experience beyond the training period, as reported above, has resulted in mixed findings. The discrepant findings may reflect discrepancies in the operational definitions of experience between studies. The most common operationalization has been years of practice examined cross-sectionally. When using this definition, Huppert et al. (2001) reported some support for an experience effect when therapists used a standardized cognitive–behavioral treatment for panic disorder. Conversely, Franklin, Abramowitz, Furr, Kalsy, and Riggs (2003), using the same operationalization, found no significant effect of therapist experience in the treatment of obsessive– compulsive disorder. Wampold and Brown (2005) also found no effect of therapist years of experience on outcome in a naturalistic managed care setting.

Long-term longitudinal approaches in research on therapist training and experience have been rare. Recently, Goldberg et al. (2016), examined how increases in the amount of time a therapist has been doing therapy and in the number of sessions a therapist has completed may affect outcomes. They found that an increase in experience had a small, but statistically significant, negative effect on outcome; on average, as therapists gained experience, their clients’ prepost outcomes diminished slightly. This study, while informing the effect of time and cumulative cases on outcome, precludes an interpretation of the effect of therapist progression through stages of training on outcome. This is particularly true because the bulk of the data used by Goldberg et al. (2016) were generated by clinicians that contributed data during only a single stage of training, and the bulk of the cases analyzed were seen by licensed clinicians. The current study is designed to extend Goldberg et al.’s (2016) investigation to examine stage of training. The smaller time intervals represented by stage of training are of particular interest as a subset of the entire dataset, as one might be most likely to find an effect of experience in the training period, as a therapist progresses from novice to licensed therapist. The current study uses data from the same setting, but with a more specific subgroup of the therapists (those with data collected during at least two levels of training) in order to examine a more specific research question—the effect of stage of training on outcome.

Another important differentiation between the current study and Goldbergetal.(2016)is the definition of client outcome.Goldberg et al. (2016) examined outcome as an effect size for a client’s total amount of change in therapy. We agree that this is an important variable, and one that we plan to investigate by stage of training. Another important client outcome variable to examine, however, is rate of client change. Previous studies have found that there are significant differences in the rate of change for clients even if there are no significant differences in the amount of change experienced by clients (see Erekson, Lambert, & Eggett, 2015). When considering stage of training, it seems reasonable to believe that there may be differences between graduate students and licensed practitioners in how quickly they are able to effect change.

Of the studies that have examined stage of training directly (discussed above), the majority have been cross-sectional, allowing for therapist experience to become entangled with individual differences (e.g., personality or theoretical orientation). The aim of the current study was to improve upon the methodological limitations of past studies and better assess the question of whether therapist training is associated with improvements in client outcomes in psychotherapy. In order to do this, our main variable of interest was stage of training rather than cumulative cases or years practicing.

All therapists included in the study were trained in PhD programs in psychology; therefore experience levels could be compared more easily given that these individuals went through a similar training timeline. Perhaps most important, therapists and clients were tracked over time, providing a longitudinal/within-subjects design. In other words, the same therapists were examined at different points in their training, and their clients’ psychological distress was tracked at each session. We are also unaware of any other study examining client rate of change by therapist stage of training over longitudinal periods in a naturalistic setting. Given the naturalistic setting, several aspects of clinical practice could not be controlled (e.g., supervision or case assignment); the use of longitudinal data versus cross-sectional data, however, as well as the clinical utility of practice-based evidence, are considerable strengths of this study. Considering the literature reviewed above and our improvements on previous methods, we hypothesized that (a) stage of training would not be associated with clientchangeintherapy,but that(b)more advanced stages of training would be associated with faster rates of client change.

[Methods, etc.]

Discussion

The current study aimed to assess whether therapist training was associated with psychotherapy client outcomes. Our first hypothesis, that there would be no association between stage of training and total amount of change in therapy, showed mixed results. At best, and when stage of training and therapist experience (cumulative cases) were included in the same model, there were no significant differences associated with either. In other words, therapists effect about the same amount of change regardless of how experienced they are or their level of training. At worst, according to models that include only a single time variable (cumulative cases or stage of training), therapists effect less change in later stages of practice. This finding is consistent with the Goldberg et al. (2016) finding that therapist experience is associated with worse outcomes, and extends the finding to stages of training.

[...]