Weak evidence that the national homicide rate spiked in 2015: The 2015 homicide rate increased above the 90% prediction interval for our model, but not more conservative intervals

Yim, Ha-Neul, Jordan R. Riddell, and Andrew P. Wheeler. 2019. “Is the Recent Increase in National Homicide Abnormal? Testing the Application of Fan Charts in Monitoring National Homicide Trends over Time.” SocArXiv. November 4. doi:10.31235/osf.io/7g32n

Abstract
Purpose: The goal of this study is to compare the increase in the 2015 national homicide rate to the historical data series and other violent crime rate changes. 
Methods: We use ARIMA models and a one-step ahead forecasting technique to predict national homicide, rape, robbery, and aggravated assault rates in the United States. Annual Uniform Crime Report data published by the Federal Bureau of Investigation are used in our analysis.
Results: The 2015 homicide rate increased above the 90% prediction interval for our model, but not more conservative intervals. Predictions intervals for other national level crime rates consistently produced correct coverage using our forecasting approach.
Conclusions: Our findings provide weak evidence that the national homicide rate spiked in 2015, though data for 2016 – 2018 do not show a continued anomalous increase in the U.S. homicide rate.

Data and code to replicate the findings can be downloaded from https://www.dropbox.com/sh/3086vtoqly5qho6/AABq_weh2LTMtBp426vhZ0EHa?dl=0


Conclusion

Media outlets reported the seemingly abrupt increase in homicide in big cities in the United
States and scholars examining homicide trends also argued the change in homicide was substantial
enough to demand scrutiny and attention in the interest of public safety (Rosenfeld et al., 2017).
Although a growing number of studies have explored the significance of the 2015 increase in homicide,
their findings are limited by a reliance on percent change or trend analysis, which have limitations in
finding the abnormal patterns in recent changes over time (Wheeler, 2016; Wheeler & Kovandzic, 2018).
This study builds upon the prior literature by using a forecasting method with an accompanying fan
chart to examine whether the 2015 homicide rate increase is significant or just a result of random
fluctuations. Past research using percent change techniques has suggested the 2015 increase in the
national homicide rate was significant. In conjunction with their findings, we offer a similar
interpretation of the homicide increase in 2015, although having arrived at the conclusion through the
use of ARIMA prediction models and fan charts – strengthening the overall conclusion that the national
homicide rate significantly increased in 2015. Our model estimates that the 2015 increase was near a 1
in 100 chance occurrence given the historical data, which we believe is sufficient to suggest that 2015
was an anomalous increase. Although such a bright line is ultimately arguable, individuals can set such a
threshold themselves to determine if such a chance occurrence is worth further investigation or any
subsequent responses.
Our comparative analysis indicates that the change in 2015 is pronounced only in homicides and
not in other violent crime. The analysis above for homicides finds that the 90% prediction intervals
failed to cover the 2015 homicide rise while observed annual rates for the other three crime types are
covered by the prediction bands. The overall temporal patterns for robbery, a crime that trends closely
with homicide, are similar to those of homicide during the predicted period; marked by a sudden decline
in the 1990s, slight fluctuations in the 2000s, and decreasing and seemingly spiking patterns in the 2010s.
Despite the relative similarity in the patterns between homicide and robbery, the homicide rise in 2015
was significant while the increase in robbery was not. In short, comparative analysis found that the
rising pattern is only significant in homicides, implying the rise in the 2015 homicide rate was abnormal
compared to other violent crimes. It is important to note here that while homicide, robbery, and
aggravated assault rates increased from 2014 to 2016 and decreased from 2016 to 2018, the rape rate
failed to decline and increased from 2013 to 2018. It is beyond the scope of our work here to explain the
reason the rape rate did not decrease after 2016 as the other violent crime rates did, but future research
can work to uncover potential causal factors.
Even though our results align with prior studies, our paper contributes to the literature on
temporal crime patterns by suggesting and demonstrating an alternative way to monitor crime trends and
annual fluctuations in crime statistics. Federal government employees have mainly used percent change
calculations in their official reports, which can result in chasing the noise and inefficient uses of
resources. As a result, policy implemented in response to such analysis may not be as effective or could
result in unintended consequences; however, the primary concern here is that percent change
calculations cause one to draw the conclusion that a policy response is necessary when it may not be
needed. For example, the government responded to the Police Executive Research Forum (PERF)
request for financial support in response to crime increases from 2004 to 2006 by funding them to hire
more police officers, increase training and provide technical assistance. However, Rosenfeld and Oliver
(2008) later found the crime increase was not substantial and just the result of external variables rather
than a lack of law enforcement officers. Similar inferences may be made based on more recent trends,
where short term fluctuations should not be viewed as so anomalous that they demand immediate
attention and investment. If practitioners or policymakers employ the suggested method in the policymaking
process, it may lead to timelier and more effective policy responses. Lawmakers and police
departments can use the forecasting method with daily, weekly, monthly, or yearly data to estimate the
number of police officers they will need to respond to a certain number of crime incidents. Academics
can do the same to produce new knowledge of crime trends without relying on a method that is biased
and volatile (percent change) or having to wait to complete a retrospective study of longer term trends
(via structural breaks).
The methods used in this study can be also be directly applied by criminal justice practitioners.
Specifically, the one-step ahead forecasting technique can be applied as part of the problem-oriented
policing approach to law enforcement crime prevention strategies. State and federal prison bureaus can
use this to estimate future prison populations and ensure they have the appropriate capacity each year.
Prosecutors, defense attorneys, and courts at any level can do the same to adequately provide counsel
and other court functions for cases in their jurisdiction. Utilizing ARIMA models instead of percent
change or trend analysis may be able to functionally improve budget and personnel appropriations.
A current limitation of applying such advice is that the FBI is often a year behind in reporting
national level homicide estimates. Thus such forecasts are too old to be effectively used for allocation
decisions, at least at the federal level. However, the utility of such forecasts to assist in anomaly
detection are still relevant. It would be easily possible for the FBI to not only release direct estimates of
homicides, but also provide such error intervals to better contextualize such ups and downs. This would
preempt overinterpreting minor fluctuations, whether it be by the media, politicians, or criminal justice
researchers.
We do not suppose prediction modelling to serve as the only method for exploring recent
changes in criminal justice related issues, for in our own work we recognize there are limitations
involving data and analysis. One of the limitations of this study is our reliance upon official UCR data.
While homicide is likely to be detected or reported, the other violent crimes could be underreported and
introduce bias into our estimates. Also, the use of national estimates of crime may cause some errors.
The national estimates of crime rates are the result of the imputation process for incomplete information
on reported crime or missing cases. The analysis results may be less valid due to the use of data different
from true figures. Such errors can lead one to misinterpret the result that homicide increase is abnormal.
Still, the forecasting and fan chart approach offers a viable alternative method, as shown by the coverage
of the prediction intervals for not only homicides but other national level violence crime estimates here.
But its application will always have limitations. The prediction intervals are only as good as the
specified model and the data it is supplied.
Another limitation is that we rely on statistical methods in which the underlying data have no
errors themselves. The same idea of a fan chart could in principle be applied to the National Crime
Victimization Survey (Rennison, 2000), although they have estimates of the prevalence of particular acts,
not an official number. Future research may attempt to build models that not only take into account
forecast error in identifying significant shifts in crime, but also sampling error as well.
Another limitation of the work is that while we can identify if an increase is abnormal or within
the prior historical changes, we cannot directly identify a cause of those changes. So while we identify
that the 2015 homicide rate has some evidence it is anomalous compared to historical patterns, we
cannot attribute it to any particular cause, such as de-policing. We do not include any structural level
variables like past work (Gaston et al., 2019; Gross & Mann, 2017; Rosenfeld, 2016; Rosenfeld &
Wallman, 2019) that tries to tease out the effect of changing drug markets, civilian concerns of police
violence, or de-policing strategies. While incorporating different covariates may help identify different
factors that contribute to macro level homicide trends, including covariates rarely increases forecasting
accuracy (Hyndman & Athanasopoulos, 2018). Since those variables often themselves have uncertainty,
they will not necessarily improve forecasts of future crime trends. Still, this technique can be used as a
first stage test as to whether some recent shock is causing homicides to increase. If recent changes fall
within historical prediction bands, it does not offer evidence that anything abnormal is currently
occurring. One should not use minor fluctuations in homicide rates (or any crimes) to ex ante attribute
those short term fluctuations to any particular exogenous shock.
Finally, we are unable to determine if the observed abnormality in the 2015 homicide rate is a
singular outlier, or part of a structural change in the homicide trend. For several possibilities, recent
increases in macro level crime trends have not been uniform, so it is not clear if the long term crime
drop beginning in the 1990’s has plateaued, or if overall crime is starting to slowly increase in an entire
reversal from the prior crime decline. Another is that different factors over time, such as the lethality of
violence (Berg, 2019), may be causing longer term trends, but are too small to be effectively detected by
this technique. It is also the case that singular cities can disproportionately contribute to national level
homicide increases (Rosenfeld, 2016).
Even if identifying an anomalous national level change, it would not directly say whether such a
shock is relevant for the entire U.S., or just a few select jurisdictions. It is the case though that similar
analysis can be conducted at the city level to identify anomalous changes (Wheeler & Kovandzic, 2018).
But city level analyses will have more error in the forecasts than national level estimates, due to the fact
that homicide is a quite rare event. Still, we believe national level analysis are still relevant, especially
given they are often used to justify particular policy responses. Given that many cities often follow
national level trends (McDowall & Loftin, 2009), it may be better to start with national level analysis
from a monitoring perspective, and when an anomaly is detected to drill further into the data to attempt
to uncover if particular cities are driving that change.
Researchers, practitioners, media outlets, and politicians should continue to monitor violent
crime rates, comparing them to one another and within the data series before making a definitive
statement on the “trend” of any type of violence. Crime analysts employed by larger police departments
have direct access to calls for service, incident, and arrest data for their city. It is becoming more
common for police departments in big cities to make their crime data publicly available (e.g.,
opendata.gov) and to update an online database each week or month, facilitating timely access by
researchers independent of the police department. Such data availability allows studies of local crime
rates, though aggregated national studies will still lag behind actual crime occurrences due to the length
of time (up to 18 months) it takes for official UCR data to be published. Given we find evidence that
homicide trends follow a random walk pattern, as do other researchers (McDowall, 2002; McDowall &
Loftin, 2005), it would suggest the overall rate may meander up-and-down for in the short run for any
particular period, and so two or three year increases are not guaranteed proof that the crime declines
observed in the prior 25 years are reversing. The fan charts we illustrate here can help identify short run
aberrations, and thus give more immediate feedback to policy makers on crime trends, without needing
to wait for years to identify long term trends.