Chapter 5: Sensitivity Analysis

Sensitivity analysis is the process of determining the effect of model parameter values on the performance of the model.

Model Parameters

Model parameters are numeric values that influence the behavior of the model, such as:

  • probability of transmission in a disease model

  • the duration of infection

  • the size distribution of households

Some example of parameters and their likely sources include disease natural history parameters:

  • Number of days latent or infectious (probability distributions) [from literature]

  • Probability of being asymptomatic when infected [literature, expert opinion]

  • Reduction in infectivity if asymptomatic [literature, expert opinion]

  • Probability of staying home when sick [survey, expert opinion]

  • Weekend community contact increase [mobility data, literature]

  • Daily number of contacts per day in households, schools, or workplaces [observational studies, reasonable guesses]

Whenever possible, it is desirable to use values based on sound theory, the scientific literature, or direct observation. The consensus opinions of experts or other stakeholders is also a valuable source for setting parameters. However, even the best sources of information often provide only a range of values for some model parameters. And some model parameters may still be unknown or unmeasurable in real systems. The calibration process discussed in the previous chapter can be used to identify reasonable values for such model parameters.

Regardless of the source of model parameters, it is often important to understand how sensitive the model is to selected values of the model parameters. Generally speaking, if small changes in a parameter lead to significant changes in the output of the model, the model is said to be sensitive to that parameter. We might also say that the parameter is sensitive for this model. If a parameter is particularly sensitive, it may be worthwhile to spend extra time and effort to obtain an accurate value for the parameter. On the other hand, if a parameter is not particularly sensitive, then it may not require further investigation of measurement. Keep in ming the purpose of the model. Some outputs may be sensitive to some parameters, while other outputs of the model may not be sensitive to the same parameters.

This chapter describes a workflow for running a sensitivity analysis on FRED models. In particular, we describe a process called local sensitivity analysis in which parameters are varied one at a time.

The Sensitivity Analysis Process

Sensitivity analysis in FRED involves running a series of jobs that systematically vary the values of selected parameters one at a time and measure the impact of each change on a selected set of output variables.

The steps in the process include:

  1. Identify the set of variables parameters

  2. Select a reference value for of the variable parameters

  3. Select a level of variation to explore for each of the parameters

  4. Identify the set of output variables of interest

  5. Run a set of jobs that varies each input parameter around its reference value

  6. Collect the resulting output values in a spreadsheet for further analysis

Reference Values

For each uncertain parameter in the model, identify a reference value. Sources of reference values include:

  • Empirical observation

  • Scientific literature

  • Expert opinion

  • Other established models

  • Reasonable estimates

Document each reference value by citing published sources or personal communications, and record whether the reference sources provide specific values or ranges.

It’s a good practice to give each reference value a ranking, for example, 1..5 where 1 is highly uncertain, and 5 is highly certain. We may want to focus additional data-gathering efforts on parameters that are both highly uncertain and highly sensitive.

FRED Sensitivity Analysis with fred_make_sa

FRED SIMS provides a command fred_make_sa that creates a workflow for performing local sensitivity analysis. The steps of the workflow are:

  1. Create a configuration file that

    • Gives the workflow a title

    • Lists the test variables with their reference value and test range

    • Lists the FRED output variables of interest

  2. Run fred_make_sa on the config file:

    • Creates a parameter file for baseline reference values

    • Creates 2 parameter files for each test variable (with high and low values)

    • Creates a script to run a job on each parameter set

    • Creates a script to collect results from all jobs into a spreadsheet for further analysis

  3. Run the job execution script

  4. Run the data collection script

  5. Analyze the results to identify the sensitivity of the model to the test variables

The output of fred_make_sa --help gives the options and a sample configuration file:

$ fred_make_sa [--help][-n runs][-m simultaneous_runs][other options]
  -f file :       the name of the configuration file. (default = sweep.config).
  -h              print out this help message.
  -m <int> :        the number of simultaneous runs to allow (default = 1)
  -n <int> :        the number of simulation replications to run (default = 20)
  -p program :      the name of the main FRED program (default = main.fred)

#################################################
# Sample FRED Sensitivity Analysis config file
#
# Lines that starts with # are ignored.
#
FRED Sensitivity Analysis Title: Influenza

Variable Parameters:
sick_day_prob = 0.5, 5%
influenza_symp = 0.67, 5%
weekend_contact_rate = 1.5, 5%
school_classroom_size = 40, 5%
office_size = 50, 5%i

Output Variables:
FRED.Tot_Deaths

#################################################

Example

The components to include in a sensitivity analysis, including which parameters to vary and which outputs to measure, are always dependent on the specific purpose of the model. That is, if a certain output variable is very important to the purpose of the model, then that output variable should be included as the target of the sensitivity analysis.

In this example, we examine the sensitivity of a simplu flu model to some of the default parameter values in the FRED configuration file. The main.fred file is shown below:

# Main program for sensitivity analysis of contact rates

simulation {
    locations = Jefferson_County_PA
    start_date = 2021-Jan-01
    end_date = 2021-May-01
}

use FRED::Influenza {
    condition = INF
    trans = 1.0
    import_count = 10
    symp_prob = symptomatic
    stay_home_prob = sick_day_prob
}

variables {
    global exposed
    exposed.output = 1
    global sick_day_prob
    global symptomatic
}

state INF.E {
    exposed = exposed + 1
}

For this illustration, we select the following input parameters:

  • sick_day_prob = probability of staying home when sick

  • symptomatic = percent of infections that results in symptoms

  • Household.contact_rate = number of potentially transmissible contacts per hour in the household

  • School.contact_rate = number of potentially transmissible contacts per hour in the school

  • Classroom.contact_rate = number of potentially transmissible contacts per hour in the classroom

  • Workplace.contact_rate = number of potentially transmissible contacts per hour in the workplace

  • Office.contact_rate = number of potentially transmissible contacts per hour in the office

  • School.partition_size = maximum number of students per classroom

  • Workplace.partition_size = maximum number of workers per office group

As an output of interest, we select:

  • FRED.exposed = a global variable set to the total number of infections in the model

Suppose we have decided to measure the effects of a 10% variation in each of the first seven parameters, and that we want to vary the assumption on classroom sizes by plus or minus 5 and the office group size by plus or minus 10. The corresponding configuration file could be called flu.config:

#################################################
# Sample FRED Sensitivity Analysis config file

FRED Sensitivity Analysis Title: flu

Variable Parameters:
sick_day_prob = 0.5 10%
symptomatic = 0.67 10%
Household.contact_rate = 0.16692 10%
School.contact_rate = 0.15828 10%
Classroom.contact_rate = 0.31657 10%
Workplace.contact_rate = 0.03653 10%
Office.contact_rate = 0.07305 10%
School.partition_size = 35 5
Workplace.partition_size = 40 10

Output Variables:
FRED.exposed

Each parameter is given a reference value (first value after =), and a variation indicator. An indicator with a percent sign (e.g. 10%) means that we want to test values that increase or decrease the reference value by the indicated percent. If no % symbol occurs, the reference valued will be varied by the absolute amount shown.

There must be at least one output variable listed.

The following bash script illustrates the typical use of fred_make_sa in a workflow.

#!/bin/bash

# script to run sensitivity analysis on influenza model

# create the scripts to run a sensitivity analysis with title “flu”
fred_make_sa -f flu.config -p main.fred -n 20 -m 5

# run the jobs in the sensitivity analysis
run_sa.flu

# collect the results in a csv file
get_results_sa.flu

In this example, we have 9 test parameters. Therefore, the fred_make_sa command creates 19 parameter files, one for the baseline case in which all parameters are set to their reference values, one for each parameter using the high value of the parameter, and one for each parameter using the low value for the parameter. These files are written in a sub-directory named SA.<title>, in this case:

$ cd SA.flu; ls
parameters.baseline
parameters.sick_day_prob=0.55
parameters.sick_day_prob=0.45
parameters.symptomatic=0.737
parameters.symptomatic=0.603
parameters.Household.contact_rate=0.183612
parameters.Household.contact_rate=0.150228
parameters.School.contact_rate=0.174108
parameters.School.contact_rate=0.142452
parameters.Classroom.contact_rate=0.348227
parameters.Classroom.contact_rate=0.284913
parameters.Workplace.contact_rate=0.040183
parameters.Workplace.contact_rate=0.032877
parameters.Office.contact_rate=0.080355
parameters.Office.contact_rate=0.065745
parameters.School.partition_size=40
parameters.School.partition_size=30
parameters.Workplace.partition_size=50
parameters.Workplace.partition_size=30

The command fred_make_sa -f flu.config -p main.fred -n 20 -m 5 also creates a script named run_sa.flu that runs a fred_job command for each of the generated parameter files using the options listed on the fred_make_sa command line. In this case, we run 20 simulations for each job, using 5 parallel executions.

Finally, fred_make_sa creates a second script called get_results_sa.flu that collects results for each output variable in a csv file showing how the variation in one of the parameters impact the output variable.

Upon completion of the 19 fred_job commands in the run_sa.flu script, the following table of results is created by get_results_sa.flu:

../_images/chapter5.results.png

Each job in the sensitivity analysis results in one row in the table. The first column shows the parameter value that was used in the job; all other parameter were set to their reference values. The columns labeled median, ci_low and ci_high give the median and confidence interval for the output variable over the 20 runs. The final column gives the percentage difference for the output variable between the job with the parameter value in the first column and the baseline job.

If there is more than one output variable, a separate spreadsheet is created for each output variable.

As the table above shows, the results show that some parameters are more sensitive than others. For example, a 10% variation in the sick_day_prob results in more than 6% change in the number of infections in the model. The parameters symptomatic and Household.contact_rate have less effect than sick_day_prob, but much more effect than the remaining parameters, which change the overall infections by less than 2%.

The interpretation of these results depends on the purpose of the model. If the model is judged to be highly sensitive to a given parameter, then it might be worthwhile to investigate other sources of data to help reduce to uncertainty in the parameter’s reference value. On the other hand, parameters that are judged to be insensitive may not require further investigation.

Further Sensitivity Analysis

In addition to the approach described here, there are many other approaches to uncertainty analysis that are beyond the scope of this guide. For example Global Sensitivity Analysis investigates how a model is affected by changing combinations of parameters, and Probabilistic Sensitivity Analysis relates the output of a model to the uncertainty distribution of the input parameters. For these approaches, the fred_sweep command may provide the basis for more sophisticated methods.