Nicholas Reich: Biostatistics and Infectious Disease Epidemiology

A modeler's primer for working with SARS-CoV-2 genomic data

Tue, 15 Feb 2022 00:00:00 +0000

SARS-CoV-2 is the name of the virus that causes COVID-19. There have been a lot of really interesting analyses of SARS-CoV-2 genomic data that look at patterns over time in the circulation of different variants. When we started writing this post, we didn’t know how to use or obtain these data, many of which are publicly available. So we wrote this primer for accessing genomic data on SARS-CoV-2 that are available for the US. These data are made public through GenBank and are pre-processed and served publicly by the Nextstrain team. We hope you find it useful!

Background on genomic data

GISAID and GenBank

Over the last two years, COVID-19 genomic data have been collected and stored in two central data repositories, GISAID and GenBank.

GISAID is a global science initiative that provides open-access genomic data of influenza viruses and the SARS-CoV-2 virus. With over 6 million submissions, GISAID currently hosts the largest repository for SARS-CoV-2 sequences. Because GISAID data are not in the public domain, GISAID data can only be accessed after creating a username and agreeing to a Database Access Agreement.

GenBank is an annotated collection of publicly available DNA sequences. It is the genetic sequence database of the US NIH. GenBank currently has over 3 million sequence read archive (SRA) runs and over 3.8 million nucleotide records for SARS-CoV-2. This database was created to provide access to the most up-to-date and comprehensive DNA sequence information, therefore there are no restrictions on the use or distribution of this data.

Though GISAID contains more variant data for SARS-CoV-2 globally, for projects interested in looking at data in the US may find that GenBank is sufficient. This is because much US sequencing is conducted through CDC contracts that stipulate a GenBank submission.

Clade and variant nomenclature

There are a number of naming systems in use for labeling variants for the SARS-CoV-2 virus.

Two naming systems relevant for tracking variants are Nextstrain and PANGO.

The Nextstrain naming system for COVID-19 involves labeling major clades by the year they first emerged, and a letter. For example, 19A was the first major clade detected in 2019.

A new major clade is named if it is at least 2 nucleotides away from its parent clade and if it reaches 20% frequency in the global population or for at least 2 months, or if it reaches 30% regional frequency for at least 2 months. A major clade may also be named if it is a variant of concern, even if it has not reached the required population frequency.

To signify clade major lineages with a hierarchical structure, emerging clades may be written as a string of parent clade lineages. For example, 19A.20A.20C for the major clade 20C. In the Nextstrain notation, emerging clades are listed with a parent clade and their defining nucleotide mutation or amino acid mutation. For example, 19A/28688C or 20B/S.484K. To note variants of concern, this nomenclature also labels them with their relevant spike mutation and a version numbering. For example, 20I/501Y.V1

Additional information regarding the original naming naming conventions using the Nextstrain naming structure can be found at the link: https://nextstrain.org/blog/2020-06-02-SARSCoV2-clade-naming and updated nomenclature can be found at the link https://nextstrain.org/blog/2021-01-06-updated-SARS-CoV-2-clade-naming

The rules of Pango lineage naming are complex. Information describing the naming rules for Pango lineages can be found here. In the Pango naming system, lineages are named to signify clusters of infections with shared ancestry and to highlight epidemiologically-relevant events.

For modelers conducting a variant-level analysis, we recommend using Nextstrain naming conventions. For modelers wishing to conduct a sub-variant analysis, Nextstrain naming conventions may not be specific enough and therefore PANGO naming conventions may be useful in these settings.

Data file types

When looking to access genomic data, raw data is stored in a number of different file types. For variant level analyses, the “metadata” file described below is a flat tab-separated dataset that includes one row per sample and has information such as the sample’s origin, cell line, and preparation method, as well as which variant was identified in the sample.

The FASTA file contains the detailed nucleic acid or amino acid sequence information in a text fie format for for a single sample.

Limitations in using genomic data

While genomic data is a potentially useful data source, there are flaws in the data that are worth noting at this time. One issue with using genomic data is that in order to capture emerging variants in a geographic region, experts recommend sequencing at least 5% of available samples. However, as of December 2021, there were 10 states in the US that had sequenced fewer than 2% of available samples. Thus, it is possible that within the US, variants and emerging strains are not being adequately detected. Information about the heterogenous testing rates in the US can be found in the Nature news article linked here. Additionally, in April of 2021, genomic data was only available for only 1.6% of all positive cases, thus it is possible that variant information before this date is inaccurate the national level (see another Nature article).

Another issue that may bias population estimates of variant prevalence is that there are not clear guidelines on how positive samples are selected for genotyping. If the more severe or more surprising cases are sent out to be genotyped, it may influences the results. This is problematic because without knowing how samples are selected for testing, we cannot control for this issue in models.

When using genomic data in models, it is important to consider factors such as sampling density, the timing of sample collection, the portion of the viral genome sequenced, quality of sequencing data and the mutation rate of the virus itself. All of these factors may impact the validity of variant prevalence values (see Villabona-Arenas et al.).

Downloading Nextstrain-curated GenBank data

The Nextstrain project makes daily snapshots of GenBank data available for the US and the world. Specifically, the flat tab-separated file available at https://data.nextstrain.org/files/ncov/open/metadata.tsv.gz is updated daily, typically around 11am or noon US Pacific time. This file is large (>350MB as of 2022-02-09). For the below, we assume that you have downloaded this file and unzipped it so the .tsv file can be read in directly.

A codebook for the fields in the dataset are available here.

Initial GenBank data exploration

Let’s start by loading some tidyverse packages that will be useful for us and then by reading in the dataset.

library(tidyverse)
library(lubridate)
theme_set(theme_bw())
genbank_global <- read_tsv("../data/metadata.tsv")

This is a large file, with 3707198 rows. For starters, we create a version of these data that contain only US data, and only retains columns we are interested in. Further, we will reformat certain columns.

us_dat <- genbank_global %>% 
  filter(country=="USA") %>%
  mutate(date = ymd(date), 
         date_submitted = ymd(date_submitted),
         reporting_lag = as.numeric(date_submitted - date)) %>%
  select(strain, virus, Nextstrain_clade,  ## info on the virus
         region, country, division, location, ## info on location
         host, sampling_strategy, ## info about the sample
         date, date_submitted, reporting_lag) ## dates

us_dat %>%
  group_by(Nextstrain_clade) %>%
  summarize(n_samples = n()) 
#> # A tibble: 25 × 2
#>    Nextstrain_clade n_samples
#>    <chr>                <int>
#>  1 19A                   4445
#>  2 19B                   3966
#>  3 20A                  50509
#>  4 20B                  23174
#>  5 20C                  57723
#>  6 20D                    613
#>  7 20E (EU1)              134
#>  8 20G                  62703
#>  9 20H (Beta, V2)        2188
#> 10 20I (Alpha, V1)     184305
#> # … with 15 more rows

Note that the sampling_strategy field is empty for all US data.

us_dat %>%
  group_by(sampling_strategy) %>%
  summarize(n_samples = n()) 
#> # A tibble: 1 × 2
#>   sampling_strategy n_samples
#>   <chr>                 <int>
#> 1 ?                   1838603

The following figure shows the reporting lags by state as computed in the data as the difference between the date the sample was taken and the date_submitted, which is the date the sample was submitted to the GenBank system. There appears to be substantial variation by location (note the shorter lags in MA, CA and VT), with 75% of samples typically reported by 1 month out. Note this is subsetting to look at all data starting in August of 2021. Some analyses, for example the R_t analysis that the Bedford Lab has done, remove all samples from the last 10 days, to try to remove small sample effects in the early reporting.

us_dat %>%
  filter(date > ymd("2021-08-01")) %>%
  ggplot(aes(x=division, y=reporting_lag)) + 
  geom_boxplot() +
  scale_y_log10(name= "lag (days, log scale)", breaks=c(7, 14, 21, 30, 90, 360, 720)) +
  xlab(NULL) +
  theme(axis.text.x = element_text(angle=60, vjust=1, hjust=1))

Plot of samples over time

us_dat %>%
  group_by(date) %>%
  summarize(n_samples = n()) %>% 
  ggplot(aes(x=date, y=n_samples)) +
  geom_bar(alpha=.3, stat="identity") +
  geom_smooth(span=.1, se=FALSE)

Here is a plot of the prevalence of each clade by week across 2021.

strains_of_interest <- 
  c("20I (Alpha, V1)", 
    "20J (Gamma, V3)", 
    "21A (Delta)",
    "21C (Epsilon)",
    "21I (Delta)",
    "21J (Delta)",
    "21K (Omicron)"
    )

by_clade <- us_dat %>%
  filter(year(date) == 2021) %>%   ## focus only on 2021 
  filter(Nextstrain_clade %in% strains_of_interest) %>%
  mutate(epiweek = epiweek(date)) %>%
  filter(epiweek < 53) %>%        ## values of 53 are at the start of 2021 
  group_by(Nextstrain_clade, epiweek) %>%
  summarize(clade_total = n()) %>%
  group_by(epiweek) %>%
  mutate(total_samples = sum(clade_total),
         pct_clade_in_epiweek = clade_total/total_samples)  

by_clade %>%
  ggplot(aes(x=epiweek, y=pct_clade_in_epiweek, color=Nextstrain_clade)) +
  geom_line() +
  xlab("epiweek in 2021") +
  ylab("% of samples by clade")

Acknowledgments

We wish to thank all the labs whose effort and time are essential to making these important data available. Also, thanks to Trevor Bedford and the Nextstrain team for providing the processed version of the metadata files and providing some pointers and insights about these data as we were writing this post.

Forecasting tools in development

Mon, 17 Jun 2019 00:00:00 +0000

As I’ve been writing up a progress report for my NIGMS R35 MIRA award, I’ve been reminded at how much of the work that we’ve been doing is focused on forecasting infrastructure. A common theme in the Reich Lab is making operational forecasts of infectious disease outbreaks. The operational aspect means that we focus on everything from developing and adapting statistical methods to be used in forecasting applications to thinking about the data science toolkit that you need to store, evaluate, and visualize forecasts. To that end, in addition to working closely with the CDC in their FluSight initiative, we’ve been doing a lot of collaborative work on new R packages and data repositories that I hope will be useful beyond the confines of our lab. Some of these projects are fully operational, used in our production flu forecasts for CDC, and some have even gone through a level of code peer review. Others are in earlier stages of development. My hope is that in putting this list out there (see below the fold) we will generate some interest (and possibly find some new open-source collaborators) for these projects.

Here is a partial list of in-progress software that we’ve been working on:

sarimaTD is an R package that serves as a wrapper to some of the ARIMA modeling functionality in the forecast R package. We found that we consistently wanted to be specifying some transformations (T) and differencing (D) in specific ways that we have found useful in modeling infectious disease time-series data, so we made it easy for us and others to use specifications.
ForecastFramework is an R package that we have collaborated on with our colleagues at the Johns Hopkins Infectious Disease Dynamics lab. We’ve blogged about this before, and we see a lot of potential in this object-oriented framework for both standardizing how datasets are specified/accessed and how models are generated. That said, there still is a long ways to go to document and make this infrastructure usable by a wide audience. The most success I’ve had using it so far was having PhD students write forecast models for a seminar I taught this spring. I used a single script that could run and score forecasts from each model, with a very simple plug-and-play interface to the models because they had been specified appropriately.
Zoltar is a new repository (in alpha-ish release right now) for forecasts that we have been working on over the last year. It was initially designed with our CDC flu forecast use-case in mind, although the forecast structure is quite general, and with predx integration on the way (see next bullet) we are hoping that this will broaden the scope of possible use cases for Zoltar. To help facilitate our and others use of Zoltar, we are working on two interfaces to the Zoltar API, zoltpy for python and zoltr for R. Check out the documentation, especially for zoltr. There is quite a bit of data available!
predx is an R package designed my colleague and friend Michael Johansson of the US Centers for Disease Control and Prevention and OutbreakScience. Evan Ray, from the Reich Lab team, has contributed to it as well. The goal of predx is to define some general classes of data for both probabilistic and point forecasts, to better standardize ways that we might want to store and operate on these data.
d3-foresight is the main engine behind our interactive forecast visualizations for flu in the US. We have also integrated it with Zoltar, so that you can view forecasts stored in Zoltar (note, kind of a long load time for that last link) using some of the basic d3-foresight functionality.

The lion’s share of the credit for all of the above are due to some combination of Matthew Cornell, Abhinav Tushar, Katie House, and Evan Ray.

Forecast Framework Demo

Tue, 29 Jan 2019 00:00:00 +0000

Want to learn how to do some forecasting with R? Here’s your chance to try out a new time-series forecasting package for R whose aim is to standardize and simplify the process of making and evaluating forecasts!

The Reich Lab uses an R package called ForecastFramework to implement forecasting models. There are many benefits to using ForecastFramework in a forecasting pipeline, including: standardized and simplified rapid model development and performance evaluation. ForecastFramework was created by Joshua Kaminsky of the Infectious Disease Dynamics Group at Johns Hopkins University. The package is open source and can be found on Github.

After watching students in the lab working on learning how to use ForecastFramework, I decided to create a step-by-step demonstration of the primary use cases of ForecastFramework. The complete demo lives at reichlab.io/forecast-framework-demos/.

As the resident CS grad-student programmer in the lab, I wanted to write these demonstrations to make ForecastFramework accessible. However, the demo only scrapes the surface of the many way that we (and others, we hope) will use ForecastFramework. We have incorporated ForecastFramework into our workflow for generating real-time dengue forecasts for the Ministry of Public Health in Thailand, and students have found it useful in generating small model comparison projects.

The online demo is separated into five sections. Each section will build off knowledge from the previous and will gradually increase in difficulty. However, the demos work as standalone scripts as well. The demos are categorized as the following:

The Data - This section will examine the raw data used in the ForecastFramework models ahead.
Defining Inputs - This section will define what an IncidenceMatrix is, show how to format your data to be used as an IncidenceMatrix, and exemplify functions of the IncidenceMatrix class.
Fitting and Forecasting - This section will focus on fitting data to a SARIMA model with ForecastFramework, using the sarimaTD package developed by Evan Ray.
Evaluating Complex Models - This section will demonstrate evaluation metrics and techniques by comparing two complex models in ForecastFramework.
Creating your own Model - This section will use object-oriented R programming demonstrate how to create your own model with ForecastFramework.

Try using ForecastFramework today! I hope you find the tutorials interesting and instructive. If you have any questions or find any bugs, please let me know! I can be found at khouse [at] umass.edu or through my personal website.

Data needs for forecasting influenza pandemics

Mon, 17 Sep 2018 00:00:00 +0000

Last week, I attended a Pandemic Influenza Exercise at the US CDC. To be clear, there is NOT a pandemic occuring right now, but the CDC ran this exercise where hundreds of staff members and outside observers and participants came together to practice going through the motions of a public health response to a major pandemic. As someone who is usually sheltered from this everyday aspect of public health decision-making, this was a fascinating window into understanding the careful, if time-pressured, scientific deliberation that underlies the response to public health emergencies.

Forecasting in a pandemic

I was sitting with a team called Risk Assessment and Forecasting, and was one of three academic forecasters in this group. (The others were Jeff Shaman and Roni Rosenfeld.) When I was invited, I wasn’t sure whether I was going to spend these three days scrambling on my computer to actually run data analyses or not. As it turned out, our job was more to listen, watch, and occasionally give some input about what kind of data would be helpful if we were needing to produce real-time pandemic forecasts. Jeff was the only one of us who had actively worked in producing real-time pandemic forecasts before (e.g., for the Ebola outbreak).

There are lots of open questions about what kinds of models would be useful and how you might go about setting up forecasting models (especially more statistical models) in a real pandemic. Fortunately for public health (although unfortuantely for the science of forecasting), we don’t get many opportunities to practice. There were a handful of good forecasting studies during Ebola, and these provide some of our best data on the models and experience of creating forecasts in real-time during large global outbreaks. Never before had a global pandemic occurred at a time when a number of different research groups had the capacity and technical knowledge to implement reasonable attempts at forecasting the trajectory of a pandemic.

Over the course of the past week, our team at the CDC exercise talked a bit about data that would be useful in real-time for pandemic forecasting. One thread of this conversation focused on real-time epidemiological data that would be collected during the outbreak. For example, consistently sampled surveillance data, especially laboratory-confirmed case counts and strain-specific positivity rates, would provide valuable measures of infection rates in a population at a given time that could be used in models.

However, a second thread focused on data that exist right now in various forms that may also be useful in pandemic settings, especially if they were made more readily and easily available. I am a forecaster and statistician whose team builds mostly “statistical” models, that is, models that rely strongly on previously observed empirical data and patterns in similar prior situations. This is an uncomfortable position to be in when forecasting new emerging pandemics, as there are so few “comparable” events on record. This approach constrasts with more “mechanistic” model-building approaches that rely heavily on a structured set of assumptions about susceptibility in the underlying population, and how transmissible the population is, etc…

So the statistician/data-scientist in me argues that there would be not insignificant value in having some curated datasets on pandemics. Certainly things have changed dramatically since 1918, 1957, 1968, and even 2009, so what we can learn from those settings in a new influenza pandemic would be limited. But having both a historical understanding of pandemics as well as some data from those outbreaks can provide useful insights into potential future dynamics of pandemics.

Some circulating questions during the recent CDC exercise revolved around whether there would be a hypothetical second wave and if so, what it would look like. Mechanistic models could start to answer that question quantitatively early on in an outbreak, but I’d argue that for a while they’d be so uncertain as to be almost useless in a hypothetical pandemic response. To start to answer the question about a second wave, I wanted to look at past data. It is not obvious what the best source of data like this is, and there does not appear to be a single authoritative repository or archive of data on pandemic flu. That’s what set me off on a hunt to make some plots of influenza pandemic dynamics over the past century.

Finding data from Project Tycho

After a brief and unsuccessful round of Googling for public datasets on pandemic flu, I turned to Project Tycho, a carefully curated public repository of infectious disease data. As it turns out (as you will see below), the data they have on influenza (and pneumonia) is not particularly helpful or exhaustive, but it was at least a place to start. So I’ll walk through an edited and simplified version of some of the back-of-the-envelope descriptive analyses I ran for myself this week. In particular, I’m going to walk through the creation of two different plots of public pandemic data from two different Project Tycho data sources that I tried to use to help me think about the potential of a second wave.

I’m going to walk through this analysis in R, so first let’s get some preliminaries out of the way to get my R session set up…

library(data.table)
library(dplyr)
library(ggplot2)
library(MMWRweek)

theme_set(theme_bw())  ## because I don't like the ggplot default theme :-)

The 1918 influenza pandemic: plotting pneumonia deaths

Let’s start by using the Project Tycho API to access data through R. To replicate this analysis, you will need to create an account (if you don’t have one already), find your API key on your profile page, and insert it in the code below.

APIKEY <- "put-your-key-here"

I toggled back and forth for a while from my code and the Project Tycho API site to build some code to construct an API query. It took some time to figure out exactly which “Condition Name” I should use, but after some hunting and pecking it turned out the best data for either pneumonia or influenza during the 1918 pandemic is case and death data for pneumonia. The following code sets up the query:

baseAPIlink <- "https://www.tycho.pitt.edu/api/query?"
ConditionName <- "Pneumonia"
CountryISO <- "US"
PeriodStartDate <- "1918-01-01"
PeriodEndDate <- "1921-12-31"

API_link <- paste0(
    baseAPIlink,
    "apikey=", APIKEY,
    "&ConditionName=", ConditionName,
    "&CountryISO=", CountryISO,
    "&Fatalities=0&PartOfCumulativeCountSeries=0",
    "&PeriodStartDate>=", PeriodStartDate,
    "&PeriodEndDate<=", PeriodEndDate
)

Then, I downloaded the data and made a few small data transformations for easier management later, like making date fields dates and expanding the dataframe to have NA for missing values instead of just having entirely missing rows.

pneumo_dat <- read.csv(API_link, header=TRUE) %>%
    mutate(
        PeriodStartDate = as.Date(PeriodStartDate),
        PeriodEndDate = as.Date(PeriodEndDate)
        ) %>%
    tidyr::complete(CityName, PeriodStartDate)

Since not all cities have a lot of cases, I did some subsetting to identify just cities that had both a lot of cases and weeks where cases were reported.

cities_with_1918 <- pneumo_dat %>% 
    filter(PeriodEndDate>as.Date("1919-12-31")) %>%
    group_by(CityName) %>%
    summarize(tot_records = sum(!is.na(CountValue))) %>%
    filter(tot_records>60) %>% .$CityName

big_cities <- pneumo_dat %>% 
    group_by(CityName) %>%
    summarize(tot_count = sum(CountValue, na.rm=TRUE)) %>%
    filter(tot_count>1000) %>% .$CityName

Once we have all that done, it’s not that difficult to use ggplot to construct a plot of pneumonia deaths during and after the 1918 pandemic.

ggplot(filter(pneumo_dat, CityName %in% big_cities, CityName %in% cities_with_1918), 
    aes(x=PeriodStartDate, y=CountValue)) +
    geom_line() + facet_grid(CityName~., scales="free_y")

The 1968 influenza pandemic: pneumonia and influenza deaths

The dataset I used above ends in the early 1950s, so it didn’t cover the other influenza pandemics of the second half of the 20th century. However, the “Level 2” data from Proejct Tycho (available, as best I could tell via a zip file download but not through the API) does have some different data that covers the 1968 pandemic. If you’re trying to run this code, you will need to download the Level2 data and insert below the path to your csv file.

fname <- "your-path-to-download/ProjectTycho_Level2_v1.1.0.csv"

I used data.table to read it in, since it’s a big ol’ file (~450MB).

DT <- fread(fname)
fludat <- DT %>%
    select(-url) %>%
    filter(disease %in% c("INFLUENZA", "PNEUMONIA", "PNEUMONIA AND INFLUENZA")) %>%
            mutate(
                from_date = as.Date(from_date),
                to_date = as.Date(to_date),
                year = as.numeric(substr(epi_week, 0, 4)),
                city_state = paste(loc, state)
            ) %>%
    tidyr::complete(city_state, epi_week)

As was necessary in the first example, this needed a bit of cleaning up to prepare for plotting.

aggpidat <- fludat %>% 
    filter(epi_week>196700, epi_week<197100, event=="DEATHS", disease=="PNEUMONIA AND INFLUENZA") %>%
    mutate(date = MMWRweek2Date(as.numeric(substr(epi_week,0,4)), as.numeric(substr(epi_week,5,6)))) 

Again, I tried to focus on cities where there were higher counts and more weeks with observed data.

big_pi_cities <- aggpidat %>% 
    group_by(city_state) %>%
    summarize(tot_count = sum(number)) %>%
    filter(tot_count>2000) %>% .$city_state

cities_with_1968 <- aggpidat %>% 
    filter(epi_week>196800, epi_week<197000) %>%
    group_by(city_state) %>%
    summarize(tot_records = n()) %>%
    filter(tot_records>60) %>% .$city_state

And, here’s the plot:

ggplot(filter(aggpidat, city_state %in% big_pi_cities, city_state %in% cities_with_1968), aes(x=date, y=number)) +
    geom_line() + facet_grid(city_state~., scales="free_y")

What is there to learn from pandemic data?

These few plots tell us just a little bit about these historical pandemics. In 1918, the best data from Tycho seemed to be about pneumonia, but even then, as is evident from the city-level data, there are lots of gaps in the observations. This is not as evident in the overall aggregate data shown in the first plot above the fold. In 1918, the data clearly show a first peak in late 1918 and then a second peak in early 1920, although most cities are missing data for almost all of 1919. In the 1968 pandemic, there seems to be a small first wave (or maybe just a larger seasonal wave) in the 1967/1968 season followed by a larger peak right around December 1968 and January 1969. But this analogy to the flu pandemic is dependent on the assumption that reported pneumonia deaths serve as a reasonable proxy for incident flu cases.

Certainly there must be more and better data out there on influenza pandemics. For the 2009 N1H1 pandemic, we could retrieve data from 2009 from the CDC FluView application (which has a nice interface from R in the cdcfluview package) But I’m sure there is more data on these older pandemics as well out there. It might require heading into the depths of library stacks, or poring through academic papers, or finding other less well-established data repositories. But having this data systematically organized in one place would be useful, and not all that hard, at least to get it started.

Once again, this is just one very small piece of a huge puzzle of pandemic response. Clearly assembling some historical data isn’t going to prevent the next pandemic, nor will it directly inform countermeasures or interventions that are part of the response. Especially given how difficult it is to generalize from these once-in-a-generation type experiences, it is unclear how directly applicable these data are to a hypothetical emerging outbreak. That said, we have precious little real data about pandemics to go on. And there’s no reason to drive blind without having better and more accessible versions of what could be a valuable source of information about dynamics of pandemic second waves, spatial variation in outbreaks, and more.

Building a collaborative ensemble to forecast influenza

Tue, 28 Nov 2017 00:00:00 +0000

In March 2017, a group of influenza forecasters who have participated in the CDC FluSight challenge in past seasons established the FluSight Network, a multi-institution and multi-disciplinary consortium of forecasting teams. This group worked throughout 2017 to create a public, real-time collaborative ensemble forecasting model that provides updated forecasts of influenza in the US each week.

Background

Every flu season since 2013/2014, the CDC has organized a challenge where, each week from early November through mid April, teams submit forecasts about the flu season in the US. For every week in a season, each submission contains forecasts for seven targets of public health interest specified by the CDC for each of the 11 HHS regions. The region-level targets are: the fraction of doctor’s visits where due to influenza-like-illness in each of the next four weeks of the season, the week of season onset, the week in which the peak wILI occurs, and the level of the peak wILI. The forecasts themselves are text files containing, in a specified format, data that encode predictive distribution for these targets.

Throughout this project, the central question has been can we provide better information to decision makers by combining forecasting models, and specifically by using past performance of the component models to inform the ensemble approach. All previous participants in FluSight challenges were invited to join the FluSight Network. Four groups decided to participate and contributed 21 models in total using a diverse array of methodologies, including kernel conditional density estimation, Bayesian state-space models, simple seasonal models, auto-regressive models for time-series, and susceptible-infectious-recovered-susceptible compartmental models, to name just a few.

Institution	No. of models	Team leaders
Delphi team at Carnegie Mellon	9	Logan Brooks, Roni Rosenfeld
Columbia University	7	Teresa Yamana, Sasikiran Kandula, Jeff Shaman
Los Alamos National Laboratory	1	Dave Osthus
Reich Lab at UMass-Amherst	4	Nicholas Reich, Abhinav Tushar, Evan Ray

Prior to the start of the 2017/2018 influenza season in the US (first submissions were due on November 6), we assembled these 21 distinct forecasting models for influenza, each with forecasts from the last seven influenza seasons in the US. (To the extent possible, these forecasts were only allowed to use data available at the time the forecasts were made.)

Subsequently, we conducted a cross-validation study to compare five different methods for combining these models into a single ensemble forecast. Specifically, this was done by leaving one season out at a time, fitting each ensemble model based on the remaining seasons’ data, and generating ensemble forecasts for each week of the left-out season. Then, we evaluated and compared the performance of the ensemble models.

Ensemble specifications

All of our ensemble models are built by taking weighted averages of the component models. We examined the performance of five different possible ensemble specifications (see table below). The “equal weights” model takes a simple average of all of the models, with no consideration of past performance. The other four approaches estimated weights for models based on past performance, using the degenerate EM algorithm.

Model	No. of weights	description
Equal weights (EW)	1	Every model gets same weight.
Constant weights (CW)	21	Every model gets a single weight, not necessarily the same.
Target-type-based weights (TTW)	42	Two sets of weights, one for seasonal targets and one for weekly wILI targets.
Target-based weights (TW)	147	Seven sets of weights, one for each target separately.
Target-and-region-based weights (TRW)	1,617	Target-based weights estimated separately for each region.

Forecast Evaluation

We measured performance by computing the average score across all targets and all relevant weeks in the last seven seasons. (Including only “relevant weeks” means, e.g., that for evaluating season onset we exclude weeks after the onset has clearly occured, because at this point, the forecasts are no longer informative.) The ensemble models generally showed better average scores than any of the component models, and there was little difference between the CW, TTW, TW, and TRW models.

For submitting in real-time in 2017-2018, we selected the ensemble model that achieved the best overall score in the cross-validation experiment over the last seven seasons. This was the target-type-based model (TTW) that assigned one set of weights to each component model for the weekly incidence targets and another set of weights for the seasonal targets (onset timing, peak timing, and peak incidence).

Slides for Ensemble Talk at MIDAS

Wed, 24 May 2017 00:00:00 +0000

Here are the slides for my presentation today at the annual MIDAS conference in Atlanta, GA. The talk summarizes recent work led by post-doc Evan Ray on creating interpretable “feature-weighted density ensembles” for infectious disease forecasting. The paper is currently under review, but the preprint is available on arXiv. Check out the 2017-2018 real-time influenza forecasts from this model available on our flusight app. And here are some slices of the feature-dependent weighting functions for predicting peak incidence for influenza in the U.S.

Here is the full slide deck for the talk:

Machine Learning and Clinical Decision Making

Sun, 14 May 2017 00:00:00 +0000

I wrote a response to Siddhartha Mukherjee’s article “A.I. vs. M.D.” that appeared in the New Yorker last month. While I submitted it as a letter to the editor, they didn’t publish it. In retrospect, perhaps it was a bit long-winded for their curt and pithy letters section. Mukherjee’s article was published on the heels of Evan submitting his latest work on improving the consistency of infectious disease prediction using interpretable model averaging methods. What follows is the letter I submitted.

Siddhartha Mukherjee nicely characterizes both sides in a looming paradigm shift in clinical practice: how to incorporate the promises of “big data” into decision-making and diagnoses that impact real people (“A.I. vs. M.D.”, April 3rd). In doing so, he exposes a rift not just between machine learners (who cling to their data and models) and clinicians (who often trust in anecdote and experience), but a rift in the machine learning community itself.

Many “deep learning” algorithms are, as Mukherjee describes, black boxes whose promise to “replace dermatologists and radiologists” would understandably rankle even very forward-thinking diagnosticians. When life-and-death decisions are being made, it is a lot to ask anyone to trust the black box.

However, not all predictive models are or need to be opaque to the experts they are designed to aid. Many predictive models (for example, certain “model averaging” approaches) can provide more information to clinicians and patients about how they work, and why they succeed and fail. Development and investigation of transparent approaches can and should be emphasized over other more opaque approaches that thwart interpretability.

While data-driven approaches deserve to make their way into clinical and public-health decision-making, the benefits for patients will only be maximized if clinicians, biostatisticians, and computer scientists partner closely. To succeed, future efforts must focus on enhancing existing intuitive, interpretable predictive models and finding ways to peel back the layers of complexity of others. With transparency will come trust, and improved clinical care.

U.S. Influenza Forecast updates (Nov 29 edition)

Wed, 30 Nov 2016 00:00:00 +0000

We updated our U.S. influenza forecasts on Tuesday, November 29th. (We tend to update the forecasts on Mondays, but the CDC data release was delayed this week due to Thanksgiving last week.) Overall, the data and the short-term forecasts for flu are showing regional circulation of flu that is a bit below the CDC-defined baseline levels. The two exceptions are in HHS Region 2 (NY and NJ) which is right at its baseline level, according to the most recent data from the CDC (reported through November 19th), and HHS Region 4 (the southeastern corner of the US) which already has risen above its baseline. Region 4 has historically had somewhat earlier seasons than the rest of the US. Check out our interactive FluSight app for more details on each region.

Reported U.S. regional influenza incidence in Nov 13-19 (MMWR week 46), 2016. Colors show percent above or below baseline:

Last week, I gave a quick under-the-hood look at how we make our forecasts. This week, I’m going to interpret more directly our forecasts for this week.

Our model is currently combiing predictions from three component models to deliver a single forecast. Depending on the region, it is combining the predictions with different weights. One thing that we’ve noticed is that the seasonal auto-regressive integrated moving average (SARIMA) model is a bit more “jumpy”, or willing to predict a rapid increase than the other models. This year, it has shown mixed performance. For example, last week it predicted upticks in Regions 2 and 4 that were almost exactly right. However, it also predicted upticks in Regions 3 and 6 that were wrong.

These are early assessments of prediction accuracy, and they may change, as the CDC often does update its data over the course of a few subsequent weeks. See for example this example of data reported in week 52 of 2015 (dark line with dots in the figure below) that was subsequently adjusted down (solid green line): This “backfill” issue (adjustment of reported data in weeks following the initial report) is not something that we have yet accounted for in our forecasts, although it is on the short-list of issues to add to our development model that will be udpated throughout the season.

In most regions, our ensemble model is sticking closely to the KCDE model for its weekly forecasts. The figure below, showing our current forecasts for Region 4, highlights how the red predictions (ensemble model) overlay the blue predictions (KCDE) almost exactly. For now, the ensemble is resisting the urge to follow the urgency of the green SARIMA model or the conservatism of the orange model which is pulling the trajectory back towards a seasonal average:

Stay tuned for periodic updates throughout the season!

Under the hood of our real-time flu predictions

Wed, 23 Nov 2016 00:00:00 +0000

For the second year in a row, the Reich Lab is participating in the CDC FluSight challenge, a project where teams from around the country submit real-time predictions of influenza to the CDC. The teams use a variety of different models and methods to generate these predictions, from an empirical Bayes method that uses Google search data to a extended Kalman-filter method that uses humidity data to our kernel conditional density estimation method using recent incidence, and there are many others!

This year, we – well, mostly Evan – have developed a new ensemble method that combines predictions from different models. We – mostly Abhinav – also created a visualizer for our predictions. Check it out here! It’s still early in the season, and we’re not seeing much data to suggest that this will be an unusually high or low year, but that’s largely because there just isn’t much information in the early-season data. In this post, I’m going to give you a quick tour under the hood of our ensemble forecasting methodology. At some point, we’ll have an article up on GitHub or arXiv, but for now, this explanation will have to suffice.

Details of the challenge

We call our team the Kernel of Truth (left over from our KCDE methods last year, although we hope the name still is appropriate). The contest is based on predicting a measure of influenza incidence that represents the percentage of all doctor’s visits that are for influenza-like-illness (ILI), weighted by population. The measure is called “weighted ILI” and its units are percentage points. Per contest rules, all submissions have to submit full predictive distributions each week from November through April for seven different targets of interest, for each of the HHS regions in the U.S. (and for the country as a whole). Here are the targets

incidence for each of the next four weeks
onset week: the first week of the first sequence of three weeks to be above the regional CDC baseline for weighted ILI
peak week: the week in which peak incidence will occur
peak incidence: the actual value of weighted ILI at the peak week

Component models

For our submissions this year, we obtain the final predictive distributions as a weighted average of predictions from three component models:

A “fixed” model using either Kernel Density Estimation (for onset week, peak week, and peak incidence), or a generalized additive model (for predictions of incidence at horizons 1 to 4 weeks). The raw predictions from this model do not change as new data are observed over the course of the season (though the predictions for incidence in individual weeks do depend on the week being predicted), and can be interpreted as a representation of “everything that we have seen in the past”. A separate model fit is obtained for each region. In week 3 of the competition (Nov 21, 2016), we modified this method to truncate the predictive distributions for onset timing, peak timing, and peak incidence so that values that have been eliminated by previously observed incidence are assigned low probability. Currently, this is done in a very ad hoc manner.
A model combining Kernel Conditional Density Estimation (KCDE) and copulas. this method is described in more detail here. In brief, KCDE is used to obtain separate predictive densities for each future week in the season. In order to predict seasonal quantities (onset, peak timing, and peak incidence), we use a copula to model dependence among those individual predicitive densities, thereby obtaining a joint predicitive density for incidence in all future weeks. Predicitive densities for the seasonal quantities can be obtained as appropriate integrals of this joint density. A separate model fit is obtained for each region.
A seasonal auto-regressive integrated moving average (SARIMA) model. This model is fit to seasonally differenced log(weighted_ili) using a stepwise procedure to select the model specification. A separate model fit is obtained for each region.

Ensemble model

The final predictions are obtained as a linear combination of the predictions from these component models using a method known as “stacking” or model averaging. The model weights depend on the week of the season in which the predictions are made. There is a lot of gnarly math and computation that we’re leaving out here, but if you’d like to see it let us know in the comments section and post some more details. We estimate the weights via gradient tree boosting, optimizing leave-one-season-out crossvalidated log scores (using the definition of log scores specified for this competition). Currently we are using the xgboost package in R to implement this, although there have been some rumblings about moving to another method, as this one is giving us some problems when the curvature of our loss function is negative. I’ll spare you the details for now.

We are submitting two variations on the ensemble model:

KoTstable is a stable version, in which we will hold fixed all details and model fits for the component models as well as the model weights throughout the season. Because this model will not be updated, we will be able to learn about model performance over the course of the season. These are the predictions currently shown on the FluSight visualizer.
KoTdev is a development version, which we will update over the course of the season. We have plans for tweaks to all three of the existing component models, the addition of new component models, and changes to computation of the model weights. This model provides a sandbox for development of new features and continuous improvement of our prediction methodology. In the first submission week, the predictions from KoTstable and KoTdev were

Summary

There is a lot more work to do on this to get it to where we want to it be, but one of the “advantages” of these challenges is that they force you to get stuff out there and just try it out. Some of the things that we are thinking about doing are improving the estimation methodology for the weights (including perhaps some kind of smoothing or regularization of the weights), adding a more mechanistic model that incorporates some biological features of flu, and incorporating the uncertainty in recent observations (as you can see in the app, there can be adjustments to reported cases, especially in the most recently reported weeks). So, there’s lots to do, and we’re hopefully just getting started.

Using the DELPHI API to access infectious disease data

Thu, 01 Sep 2016 00:00:00 +0000

This week I attended a workshop at the CDC about last year’s FluSight challenge, a competition that scores weekly real-time predictions about the course of the influenza season. They are planning another round this year and are hoping to increase the number of teams particiating. Stay tuned to this site for more info.

At the workshop, I learned about DELPHI’s real-time epidemiological data API. The API is linked to various data sources on influenza and dengue, including US CDC flu data, Google Flu Trends, and Wikipedia data. There is some documentation and minimal examples, and this post documents a more robust and complete example for using the API via R. I’ll note that the CDC’s influenza data, can also be accessed via the cdcfluview R package, which I’m not going to discuss here and I will focus here on accessing some of the other data sources. Here’s a teaser of this data that you can also interactively explore on the DELPHI EpiVis website:

Setup

Let’s start by loading the R script containing the relevant methods needed to access the API.

source("https://raw.githubusercontent.com/undefx/delphi-epidata/master/code/delphi_epidata.R")

Then, load in two packages that we will use to tidy and plot the data.

library(MMWRweek)
library(ggplot2)

Dengue data from Taiwan CDC

Here is some code that pulls data from Taiwan’s NIDSS, specifically asking for nationwide data, and data from the central region. A complete list of regions and locations are available. Also, I’ve specified a range of weeks from the first week of 2010 (201001) to the last week of 2016 (201653).

res <- Epidata$nidss.dengue(locations = list('nationwide', 'central'), 
                            epiweeks = list(Epidata$range(201001, 201653)))

The above command should pull data down into your current session, but it will be a little bit ‘list-y’, so here is some code I wrote to clean it up and make it a bit more of a workable dataset in R.

df <- data.frame(matrix(unlist(res$epidata), nrow=length(res$epidata), byrow=T))
colnames(df) <- names(res$epidata[[1]])[!is.null(res$epidata[[1]])]
df$count <- as.numeric(as.character(df$count))
df$year <- as.numeric(substr(df$epiweek, 0, 4))
df$week <- as.numeric(substr(df$epiweek, 5, 6))
df$date <- MMWRweek2Date(MMWRyear = df$year, MMWRweek = df$week)

Note the use of the MMWRweek2Date() function that gives us a date column in our data frame. And here is a plot of the resulting data.

ggplot(df, aes(x=date, y=count, color=location)) + geom_point() + scale_y_log10()

Wikipedia data

Let’s try loading some of the Wikipedia data on influenza and other related terms. The article. I think this reflects the number of hits on pages of certain articles, although I’m not sure.

res <- Epidata$wiki(articles=list("influenza", "common_cold", "cough"),
                    epiweeks=list(Epidata$range(201101, 201553)))
df <- data.frame(matrix(unlist(res$epidata), nrow=length(res$epidata), byrow=T))
colnames(df) <- names(res$epidata[[1]])[!is.null(res$epidata[[1]])]
df$count <- as.numeric(as.character(df$count))
df$year <- as.numeric(substr(df$epiweek, 0, 4))
df$week <- as.numeric(substr(df$epiweek, 5, 6))
df$date <- MMWRweek2Date(MMWRyear = df$year, MMWRweek = df$week)
ggplot(df, aes(x=date, y=count, color=article)) + 
  geom_point() + 
  geom_smooth(span=.1, se=FALSE)

Happy data exploring!

UPDATE: (2 Sept 2016) Roni Rosenfeld, the head of the DELPHI group at CMU, pointed out and asked me to mention that David Farrow was the force behind the creation of the epidata API and the epivis tool.