Nudges: Our megastudy with 689,693 Walmart pharmacy customers demonstrates that text-based reminders can encourage pharmacy vaccination

A 680,000-person megastudy of nudges to encourage vaccination in pharmacies. Katherine L. Milkman et al. Proceedings of the National Academy of Sciences, February 8, 2022 119 (6) e2115126119; https://doi.org/10.1073/pnas.2115126119

Significance: Encouraging vaccination is a pressing policy problem. Our megastudy with 689,693 Walmart pharmacy customers demonstrates that text-based reminders can encourage pharmacy vaccination and establishes what kinds of messages work best. We tested 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination. Reminder texts increased vaccination rates by an average of 2.0 percentage points (6.8%) over a business-as-usual control condition. The most-effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top-performing intervention included two texts 3 d apart and stated that a vaccine was “waiting for you.” Forecasters failed to anticipate that this would be the best-performing treatment, underscoring the value of testing.

Abstract: Encouraging vaccination is a pressing policy problem. To assess whether text-based reminders can encourage pharmacy vaccination and what kinds of messages work best, we conducted a megastudy. We randomly assigned 689,693 Walmart pharmacy patients to receive one of 22 different text reminders using a variety of different behavioral science principles to nudge flu vaccination or to a business-as-usual control condition that received no messages. We found that the reminder texts that we tested increased pharmacy vaccination rates by an average of 2.0 percentage points, or 6.8%, over a 3-mo follow-up period. The most-effective messages reminded patients that a flu shot was waiting for them and delivered reminders on multiple days. The top-performing intervention included two texts delivered 3 d apart and communicated to patients that a vaccine was “waiting for you.” Neither experts nor lay people anticipated that this would be the best-performing treatment, underscoring the value of simultaneously testing many different nudges in a highly powered megastudy.

Keywords: vaccinationCOVID-19nudgeinfluenzafield experiment

Discussion

The results of this megastudy suggest that pharmacies can increase flu vaccination rates by sending behaviorally informed, text-based reminders to their patients. Further, although all interventions tested increased vaccination rates, there was significant variability in their effectiveness. Attribute analyses suggest that the most-impactful interventions employed messages sent on multiple days and conveyed that a flu vaccine was already waiting for the patient. These insights from our megastudy of flu vaccinations can potentially inform efforts by pharmacies and hopefully also providers and governments around the world in the ongoing campaign to encourage full vaccination against COVID-19.

The first- and second-best-performing messages in our megastudy repeatedly reminded patients to get a vaccine and stated that a flu shot was “waiting for you.”§ This aligns with prior research suggesting multiple reminders can help encourage healthy decisions (13). In terms of message content, communicating that a vaccine is “waiting for you” may increase the perceived value of vaccines, in accord with research on the endowment effect showing that we value things more if we feel they already belong to us (33). Further, because defaults convey implicit recommendations, this message may imply that the pharmacy is recommending vaccination (34)—otherwise, why would they have allocated a dose to you? Relatedly, this phrasing implies that the patient already agrees that getting the vaccine is desirable. Finally, saying a vaccine is “waiting for you” may suggest that getting a vaccine will be fast and easy.

Remarkably, the three top-performing text messages in a different megastudy, which included different intervention messages encouraging patients to get vaccines at an upcoming doctor’s visit, similarly conveyed to patients that a vaccine was “reserved for you.” (8) The Walmart megastudy tested a wide range of different messaging tactics designed by different researchers to explore largely different hypotheses such that the two megastudies had very limited overlap in their content. Further, our outcome variable here was electing to get a vaccine in a Walmart pharmacy, whereas the prior megastudy examined whether patients accepted a proffered vaccine in a doctor’s office. Despite differences in the messenger (pharmacy versus primary care provider), the outcome (seeking a vaccine at a pharmacy versus passively accepting one in a doctor’s office), and the messages actually tested, and despite analyzing our data without incorporating any priors, the results of the two megastudies converged, painting a consistent picture of what works. Together, this evidence suggests that sending reminder messages conveying that vaccines are “reserved” or “waiting” for patients is an especially effective communications strategy in the two most-common vaccination settings in the United States (16). While more research is needed to establish boundary conditions, it is notable that a follow-up study that built on the findings from these two megastudies and adapted the best-performing treatment to nudge COVID-19 vaccinations in the winter of 2021 found convergent results (9).

The scientists who designed interventions in our megastudy were not able to accurately forecast the average or relative performance of text message interventions designed to boost vaccinations. In contrast, lay survey respondents made fairly accurate predictions of both. This contradicts past research showing that experts are generally better at predicting the effectiveness of behavioral interventions than nonexperts (35). One possible explanation is that direct involvement in the design of interventions led to bias. Another possibility is that nonexperts are generally more-accurate relative forecasters in this particular field setting. Regardless, the inability of either scientists or laypeople to anticipate the top-performing intervention underscores the value of empirical testing when seeking the best policy.

The strengths of this megastudy include its massive, national sample, statistical power, an objective measure of vaccination obtained longitudinally over a 3-mo follow-up period, and simultaneous comparison of many different interventions. Several important weaknesses are also worth noting. First, we were only able to measure the impact of our interventions on vaccinations received at Walmart pharmacies. We cannot assess whether interventions increased vaccinations in other locations or, in fact, crowded out vaccinations in other settings. Past research found no meaningful evidence of crowd-in or crowd-out from reminder messages nudging vaccine adoption (4), but we cannot rule out either possibility. Another limitation is that the population studied only included patients who had received an influenza vaccine at a Walmart pharmacy in the prior flu season and agreed to receive SMS communications from Walmart pharmacy. Even so, our key findings are consistent with a different megastudy nudging vaccination at doctors’ visits (8), in which results were not moderated by prior vaccination status. It would be ideal to replicate and extend this megastudy with patients who have no prior history of vaccination. Relatedly, while we did not observe meaningful heterogeneity in treatment effects by available demographics, future research exploring heterogeneous treatment effects with a richer set of individual difference variables would be valuable.