# Model file examples

#### David Palma, Stephane Hess

#### 2024-06-03

## About the package

Apollo is an R package designed for estimation and analysis of choice models (Train, 2008). This package allows estimating Multinomial logit (MNL), Nested logit (NL), cross-nested logit (CNL), exploded logit (EL), ordered logit (OL), Integrated Choice and Latent Variable (ICLV, Ben-Akiva et al. 2002), Multiple Discrete-Continuous Extreme Value (MDCEV, Bhat 2008), nested MDCEV (MDCNEV, Pinjari and Bhat 2010), and Decision Field Theory (DFT, Hancock and Hess 2018) models. All models support both continuous and discrete mixing (e.g. continuous random parameters, latent classes or finite mixtures), both between and within individuals (i.e at the individual and observation level). Different models can be easily combined for joint estimation. The package allows for classical estimation (i.e. maximum likelihood) as well as Bayesian estimation (i.e. hierarchical Bayes, through package RSGHB).

All functionalities are described in the manual, available at www.ApolloChoiceModelling.com. Examples can also be found on the website.

## Recommended workflow

The recommended way to use Apollo is to create a different R script for each model to estimate. These scripts are refer to as model files and contain all the necessary information to estimate the model (except for the data), and can also include post-estimation analysis of the results.

A model file should be structured around the following sections:

1. Definition of core settings: In this section the Apollo library is loaded, and global variables such as the name of the model are defined.

2. Data loading: In this section the database is loaded. Any transformation to the data that does not depend on parameters to be estimated (e.g. averages of explanatory variables) should be performed here and stored inside the database object.

3. Parameter definition: In this section, a vector containing the names and starting values of parameters is defined, as well as details of their mixing distributions if present.

4. Input validation: In this section, all inputs prepared in the previous sections are validated and stored together.

5. Likelihood definition: In this section, the user must write a function that calculates the model likelihood.

6. Model estimation and reporting: In this section, the model is estimated and the results are displayed.

7. Postprocessing of results: This is an optional section where the user can do forecasting and other post-processing of the estrimated model.

In this document, we present two example model files: a multinomial logit (MNL) model, and a mixed multinomial logit (MMNL) model. More examples, discussed in more detail are available at www.ApolloChoiceModelling.com.

## MNL model file example

In this section, we present code to estimate an MNL model using the synthetic data included in the package. In the postprocessing, we predict the impact on mode share of a 10% increase in the cost of the train fare. The utility functions of alternatives are defined as follows.

\[U_{nsi} = asc_i + \beta_{tt}tt_{nsi} + \beta_{c}*cost_{nsi} + \varepsilon_{nsi}\]

Where n indexes individuals, s choice scenarios, and i alternatives. \(asc_i\) is the alternative specific constant, \(tt_{nsi}\) is the travel time and \(cost_{nsi}\) is the cost. \(\varepsilon_{nsi}\) is an independent identically distributed standard Gumbel error term. \(asc_i\), \(\beta_{tt}\) and \(\beta_{c}\) are parameters to be estimated.

The likelihood function of this model for individual \(n\) is as follows.

\[L_{n}=\prod_{s}P_{nsi}\]

Where \[P_{nsi}=\frac{e^{V_{nsi}}}{\sum_{j}e^{V_{nsj}}}\] And \(V_{nsi}=U_{nsi}-\varepsilon_{nsi}\), i.e. the deterministic part of the utility.

The likelihood function of the MNL model is pre-coded in `Apollo`, so we do not need to type it ourselves. However, if the user prefers to write the likelihood themselves, they can also do it. The pre-coded MNL likelihood function (`apollo_mnl`) requires a series of inputs defined inside the `mnl_settings` object.

    # ####################################################### #
    #### 1. Definition of core settings
    # ####################################################### #

    ### Clear memory
    rm(list = ls())

    ### Load libraries
    library(apollo)
    #>
    #>
    #>              . ,,
    #>             ,      ,,
    #>  ,,,,,,    ,         ,,
    #> ,     ,,  ,            ,,,,.
    #> ,,     , ,,   ,,,,,,    ,,,                                 //  //
    #>   ,     ,,,.   ,,,,,.   ,,      ////                        //  //
    #> ,,     ,,,,,.           ,,     // //     //////    /////    //  //    /////
    #> ,,,        ,,           ,      //  //    /    //  //   //   //  //   //   //
    #>               ,,       ,      ////////   /    //  //   //   //  //   //   //
    #>                 ,,   ,,      //     //   /   ///  //   //   //  //   //   //
    #>                    ,         //      //  /////      ///      //  //    ///
    #>                                          //
    #>                                          //
    #>
    #> Apollo 0.3.3
    #> http://www.ApolloChoiceModelling.com
    #> See url for a detailed manual, examples and a user forum.
    #> Sign up to the user forum to receive updates on new releases.
    #>
    #> Please cite Apollo in all written material you produce:
    #> Hess S, Palma D (2019). "Apollo: a flexible, powerful and customisable
    #> freeware package for choice model estimation and application." Journal
    #> of Choice Modelling, 32. doi:10.1016/j.jocm.2019.100170

    ### Initialise code
    apollo_initialise()

    ### Set core controls
    apollo_control = list(
      modelName  ="MNL",
      modelDescr ="Simple MNL model on mode choice SP data",
      indivID    ="ID"
    )

    # ####################################################### #
    #### 2. Data loading                                   ####
    # ####################################################### #

    data("apollo_modeChoiceData", package="apollo")
    database = apollo_modeChoiceData
    rm(apollo_modeChoiceData)

    ### Use only SP data
    database = subset(database,database$SP==1)

    # ####################################################### #
    #### 3. Parameter definition                           ####
    # ####################################################### #

    ### Vector of parameters, including any that are kept fixed
    ### during estimation
    apollo_beta=c(asc_car  = 0,
                  asc_bus  = 0,
                  asc_air  = 0,
                  asc_rail = 0,
                  b_tt_car = 0,
                  b_tt_bus = 0,
                  b_tt_air = 0,
                  b_tt_rail= 0,
                  b_c      = 0)

    ### Vector with names (in quotes) of parameters to be
    ###  kept fixed at their starting value in apollo_beta.
    ### Use apollo_beta_fixed = c() for no fixed parameters.
    apollo_fixed = c("asc_car")

    # ####################################################### #
    #### 4. Input validation                               ####
    # ####################################################### #

    apollo_inputs = apollo_validateInputs()
    #> Several observations per individual detected based on the value of ID.
    #>   Setting panelData in apollo_control set to TRUE.
    #> All checks on apollo_control completed.
    #> All checks on database completed.

    # ####################################################### #
    #### 5. Likelihood definition                          ####
    # ####################################################### #

    apollo_probabilities=function(apollo_beta, apollo_inputs,
                                  functionality="estimate"){

      ### Attach inputs and detach after function exit
      apollo_attach(apollo_beta, apollo_inputs)
      on.exit(apollo_detach(apollo_beta, apollo_inputs))

      ### Create list of probabilities P
      P = list()

      ### List of utilities: these must use the same names as
      ### in mnl_settings, order is irrelevant.
      V = list()
      V[['car']] = asc_car + b_tt_car *time_car + b_c*cost_car
      V[['bus']] = asc_bus + b_tt_bus *time_bus + b_c*cost_bus
      V[['air']] = asc_air + b_tt_air *time_air + b_c*cost_air
      V[['rail']]= asc_rail+ b_tt_rail*time_rail+ b_c*cost_rail

      ### Define settings for MNL model component
      mnl_settings = list(
        alternatives  = c(car=1, bus=2, air=3, rail=4),
        avail         = list(car=av_car, bus=av_bus,
                             air=av_air, rail=av_rail),
        choiceVar     = choice,
        V             = V
      )

      ### Compute probabilities using MNL model
      P[['model']] = apollo_mnl(mnl_settings, functionality)

      ### Take product across observation for same individual
      P = apollo_panelProd(P, apollo_inputs, functionality)

      ### Prepare and return outputs of function
      P = apollo_prepareProb(P, apollo_inputs, functionality)
      return(P)
    }

    # ####################################################### #
    #### 6. Model estimation and reporting                 ####
    # ####################################################### #

    model = apollo_estimate(apollo_beta, apollo_fixed,
                            apollo_probabilities,
                            apollo_inputs,
                            list(writeIter=FALSE))
    #> Preparing user-defined functions.
    #>
    #> Testing likelihood function...
    #>
    #> Overview of choices for MNL model component :
    #>                                      car     bus     air    rail
    #> Times available                  5446.00 6314.00 5264.00 6118.00
    #> Times chosen                     1946.00  358.00 1522.00 3174.00
    #> Percentage chosen overall          27.80    5.11   21.74   45.34
    #> Percentage chosen when available   35.73    5.67   28.91   51.88
    #>
    #>
    #> Pre-processing likelihood function...
    #>
    #> Testing influence of parameters
    #> Starting main estimation
    #>
    #> BGW using analytic model derivatives supplied by caller...
    #>
    #>
    #>     it    nf     F            RELDF    PRELDF    RELDX    MODEL stppar
    #>      0     1 8.196020532e+03
    #>      1     4 7.185529562e+03 1.233e-01 1.357e-01 1.00e+00   G   3.86e+00
    #>      2     5 6.432806585e+03 1.048e-01 1.280e-01 5.09e-01   G   8.17e-01
    #>      3     7 5.934539400e+03 7.746e-02 7.042e-02 5.33e-01   S   3.05e-02
    #>      4     8 5.820228979e+03 1.926e-02 2.037e-02 3.49e-01   G   9.43e-03
    #>      5     9 5.802171727e+03 3.102e-03 2.895e-03 1.99e-01   S   0.00e+00
    #>      6    10 5.802023008e+03 2.563e-05 2.526e-05 2.02e-02   S   0.00e+00
    #>      7    11 5.802022829e+03 3.094e-08 2.874e-08 3.45e-04   S   0.00e+00
    #>      8    12 5.802022826e+03 3.757e-10 3.623e-10 2.30e-05   S   0.00e+00
    #>      9    13 5.802022826e+03 1.499e-12 1.453e-12 3.34e-06   S   0.00e+00
    #>
    #> ***** Relative function convergence *****
    #> Additional convergence test using scaled estimation. Parameters will be
    #>   scaled by their current estimates and additional iterations will be
    #>   performed.
    #>
    #> BGW using analytic model derivatives supplied by caller...
    #>
    #>
    #>     it    nf     F            RELDF    PRELDF    RELDX    MODEL stppar
    #>      0     1 5.802022826e+03
    #>      1     2 5.802022826e+03 5.016e-15 3.073e-15 8.42e-05   G   0.00e+00
    #>
    #> ***** Relative function convergence *****
    #>
    #> Estimated parameters with approximate standard errors from BHHH matrix:
    #>              Estimate     BHHH se BHH t-ratio (0)
    #> asc_car      0.000000          NA              NA
    #> asc_bus      0.011525    0.534638         0.02156
    #> asc_air     -0.649528    0.268497        -2.41912
    #> asc_rail    -1.235739    0.320460        -3.85614
    #> b_tt_car    -0.010061  6.2164e-04       -16.18480
    #> b_tt_bus    -0.016422    0.001440       -11.40473
    #> b_tt_air    -0.011831    0.002360        -5.01344
    #> b_tt_rail   -0.004779    0.001690        -2.82867
    #> b_c         -0.052923    0.001394       -37.95213
    #>
    #> Final LL: -5802.0228
    #>
    #> Calculating log-likelihood at equal shares (LL(0)) for applicable
    #>   models...
    #> Calculating log-likelihood at observed shares from estimation data
    #>   (LL(c)) for applicable models...
    #> Calculating LL of each model component...
    #> Calculating other model fit measures
    #> Computing covariance matrix using numerical jacobian of analytical
    #>   gradient.
    #>  0%....25%....50%....75%.100%
    #> Negative definite Hessian with maximum eigenvalue: -3.304563
    #> Computing score matrix...
    #>
    #> Your model was estimated using the BGW algorithm. Please acknowledge
    #>   this by citing Bunch et al. (1993) - DOI 10.1145/151271.151279

    apollo_modelOutput(model)
    #> Model run by stephane.hess using Apollo 0.3.3 on R 4.4.0 for Darwin.
    #> Please acknowledge the use of Apollo by citing Hess & Palma (2019)
    #>   DOI 10.1016/j.jocm.2019.100170
    #>   www.ApolloChoiceModelling.com
    #>
    #> Model name                                  : MNL
    #> Model description                           : Simple MNL model on mode choice SP data
    #> Model run at                                : 2024-06-03 17:49:35.951814
    #> Estimation method                           : bgw
    #> Model diagnosis                             : Relative function convergence
    #> Optimisation diagnosis                      : Maximum found
    #>      hessian properties                     : Negative definite
    #>      maximum eigenvalue                     : -3.304563
    #>      reciprocal of condition number         : 2.93236e-08
    #> Number of individuals                       : 500
    #> Number of rows in database                  : 7000
    #> Number of modelled outcomes                 : 7000
    #>
    #> Number of cores used                        :  1
    #> Model without mixing
    #>
    #> LL(start)                                   : -8196.02
    #> LL at equal shares, LL(0)                   : -8196.02
    #> LL at observed shares, LL(C)                : -6706.94
    #> LL(final)                                   : -5802.02
    #> Rho-squared vs equal shares                  :  0.2921
    #> Adj.Rho-squared vs equal shares              :  0.2911
    #> Rho-squared vs observed shares               :  0.1349
    #> Adj.Rho-squared vs observed shares           :  0.1342
    #> AIC                                         :  11620.05
    #> BIC                                         :  11674.87
    #>
    #> Estimated parameters                        : 8
    #> Time taken (hh:mm:ss)                       :  00:00:1
    #>      pre-estimation                         :  00:00:0.28
    #>      estimation                             :  00:00:0.24
    #>           initial estimation                :  00:00:0.21
    #>           estimation after rescaling        :  00:00:0.02
    #>      post-estimation                        :  00:00:0.48
    #> Iterations                                  :  10
    #>      initial estimation                     :  9
    #>      estimation after rescaling             :  1
    #>
    #> Unconstrained optimisation.
    #>
    #> Estimates:
    #>              Estimate        s.e.   t.rat.(0)    Rob.s.e. Rob.t.rat.(0)
    #> asc_car      0.000000          NA          NA          NA            NA
    #> asc_bus      0.011525    0.540840     0.02131    0.541641       0.02128
    #> asc_air     -0.649528    0.269903    -2.40652    0.266912      -2.43349
    #> asc_rail    -1.235739    0.321846    -3.83954    0.312776      -3.95088
    #> b_tt_car    -0.010061  6.3867e-04   -15.75317  6.5823e-04     -15.28521
    #> b_tt_bus    -0.016422    0.001453   -11.30287    0.001480     -11.09956
    #> b_tt_air    -0.011831    0.002402    -4.92629    0.002370      -4.99140
    #> b_tt_rail   -0.004779    0.001650    -2.89724    0.001623      -2.94420
    #> b_c         -0.052923    0.001422   -37.21134    0.001701     -31.11205

    #apollo_saveOutput(model)

    # ####################################################### #
    #### 7. Postprocessing of results                      ####
    # ####################################################### #

    ### Use the estimated model to make predictions
    predictions_base = apollo_prediction(model,
                                         apollo_probabilities,
                                         apollo_inputs)
    #> Running predictions from model using parameter estimates...
    #> Prediction at user provided parameters
    #>               car    bus     air    rail
    #> Aggregate 1946.00 358.00 1522.00 3174.00
    #> Average      0.28   0.05    0.22    0.45
    #>
    #> The output from apollo_prediction is a matrix containing the
    #>   predictions at the estimated values.

    ### Now imagine the cost for rail increases by 10%
    ### and predict again
    database$cost_rail = 1.1*database$cost_rail
    apollo_inputs   = apollo_validateInputs()
    #> Several observations per individual detected based on the value of ID.
    #>   Setting panelData in apollo_control set to TRUE.
    #> All checks on apollo_control completed.
    #> All checks on database completed.

    predictions_new = apollo_prediction(model,
                                        apollo_probabilities,
                                        apollo_inputs)
    #> Running predictions from model using parameter estimates...
    #> Prediction at user provided parameters
    #>               car    bus     air    rail
    #> Aggregate 2132.59 399.33 1645.75 2822.34
    #> Average      0.30   0.06    0.24    0.40
    #>
    #> The output from apollo_prediction is a matrix containing the
    #>   predictions at the estimated values.

    ### Compare predictions
    change=(predictions_new-predictions_base)/predictions_base
    ### Not interested in chosen alternative now,
    ### so drop last column
    change=change[,-ncol(change)]
    ### Summary of changes (possible presence of NAs due to
    ### unavailable alternatives)
    summary(change)
    #>        ID     Observation      car              bus              air
    #>  Min.   :0   Min.   :0    Min.   :0.0000   Min.   :0.0000   Min.   :0.0000
    #>  1st Qu.:0   1st Qu.:0    1st Qu.:0.0725   1st Qu.:0.0738   1st Qu.:0.0704
    #>  Median :0   Median :0    Median :0.1168   Median :0.1218   Median :0.1100
    #>  Mean   :0   Mean   :0    Mean   :0.1105   Mean   :0.1225   Mean   :0.1121
    #>  3rd Qu.:0   3rd Qu.:0    3rd Qu.:0.1509   3rd Qu.:0.1674   3rd Qu.:0.1517
    #>  Max.   :0   Max.   :0    Max.   :0.2339   Max.   :0.4326   Max.   :0.3677
    #>                           NA's   :1554     NA's   :686      NA's   :1736
    #>       rail
    #>  Min.   :-0.3028
    #>  1st Qu.:-0.2060
    #>  Median :-0.1340
    #>  Mean   :-0.1434
    #>  3rd Qu.:-0.0723
    #>  Max.   :-0.0022
    #>  NA's   :882

## MMNL model file example

In this section, we present code to estimate a mixed MNL model (MMNL) using the synthetic data included in the package. After estimation, we predict the effect of a 10% increase in the train fares. The utility function of the model remains the same than in the previous example, i.e.:

\[U_{nsi} = asc_i + \beta_{tt}tt_{nsi} + \beta_{c}*cost_{nsi} + \varepsilon_{nsi}\]

Where n indexes individuals, s choice scenarios, and i alternatives. \(asc_i\) is the alternative specific constant, \(tt_{nsi}\) is the travel time and \(cost_{nsi}\) is the cost. \(\varepsilon_{nsi}\) is an independent identically distributed standard Gumbel error term. \(\beta_{tt}\) follows a log-normal distribution with the underlying normal having a mean \(\mu_{tt}\) and standard deviation \(\sigma_{tt}\). \(asc_i\), \(\beta_{c}\), \(\mu_{tt}\) and \(\sigma_{tt}\) are parameters to be estimated.

The likelihood function of this model for individual \(n\) is as follows.

\[L_{n}=\int_{\beta_{tt}}\prod_{s}P_{nsi}f(\beta_{tt})d\beta_{tt}\]

Where \(P_{nsi}=\frac{e^{V_{nsi}}}{\sum_{j}e^{V_{nsj}}}\), \(V_{nsi}=U_{nsi}-\varepsilon_{nsi}\), and \(f\) is the probability density function of \(\beta_{tt}\). As this function does not have an analytical closed form, we estimate it using Monte Carlo integration, i.e.:

\[L_{n}\approx\frac{1}{R}\sum_{\beta_{tt}^r}\prod_{s}P_{nsi}^r\] Where \(P_{nsi}^r=P_{nsi}(\beta_{tt}^r)\), with \(\beta_{tt}^r\) a random draw of \(\beta_{tt}\) from its distribution \(f\), and R is a big number.

The code is very similar to the previous example, with only sections 1 and 3 changing. * In section 1 we set `mixing = TRUE` inside apollo_control, and we set `nCores = 2` to speed up estimation by using two computing threads (this is not mandatory). * In section 3 we define the mean (`b_tt_mu`) and standar deviation (`b_tt_sigma`) of the underlying normal distribution. We then define the type, name and number of draws used. Finally, we construct the random coefficient \(\beta_{tt}\) inside a function called `apollo_randCoeff`. We use 500 inter-individual draws that come from a standard normal distribution, which we later transform into log-normals inside `apollo_randCoeff`.

Even though in this case we only use inter-individual draws, note that inter and intra-individuals draws can be used simultaneously. Inter-individual draws capture variability between individuals, while intra-individual draws capture variability within individuals. In terms of the Monte Carlo integration, inter-individual draws are common for all observations from the same individual, while intra-individual draws are different for each observations. In terms of the likelihood function, the use of intra-individual draws would lead to \(L_{n}\approx\prod_{s}\frac{1}{R}\sum_{\beta_{tt}^r}P_{nsi}\), which is not the case in this model.

Estimation of models with mixing is computationally more demanding than models without mixing. Furthermore, using both inter and intra-individual requires large amounts of memory, which can further slow the estimation process. For this reason, this example is not run automatically. Yet, the users may copy and paste the code in a script, and run it themselves.

    # ####################################################### #
    #### 1. Definition of core settings
    # ####################################################### #

    ### Clear memory
    rm(list = ls())

    ### Load libraries
    library(apollo)

    ### Initialise code
    apollo_initialise()

    ### Set core controls
    apollo_control = list(
      modelName  ="MMNL",
      modelDescr ="Simple MMNL model on mode choice SP data",
      indivID    ="ID",
      mixing     = TRUE,
      nCores     = 2
    )

    # ####################################################### #
    #### 2. Data loading                                   ####
    # ####################################################### #

    data("apollo_modeChoiceData", package="apollo")
    database = apollo_modeChoiceData
    rm(apollo_modeChoiceData)

    ### Use only SP data
    database = subset(database,database$SP==1)

    ### Create new variable with average income
    database$mean_income = mean(database$income)

    # ####################################################### #
    #### 3. Parameter definition                           ####
    # ####################################################### #

    ### Vector of parameters, including any that are kept fixed
    ### during estimation
    apollo_beta=c(asc_car  = 0,
                  asc_bus  =-2,
                  asc_air  =-1,
                  asc_rail =-1,
                  mu_tt    =-4,
                  sigma_tt = 0,
                  b_c      = 0)

    ### Vector with names (in quotes) of parameters to be
    ###  kept fixed at their starting value in apollo_beta.
    ### Use apollo_beta_fixed = c() for no fixed parameters.
    apollo_fixed = c("asc_car")

    ### Set parameters for generating draws
    apollo_draws = list(
      interDrawsType = "halton",
      interNDraws    = 500,
      interUnifDraws = c(),
      interNormDraws = c("draws_tt")
    )

    ### Create random parameters
    apollo_randCoeff = function(apollo_beta, apollo_inputs){
      randcoeff = list()

      randcoeff[["b_tt"]] = -exp(mu_tt + sigma_tt*draws_tt)

      return(randcoeff)
    }

    # ####################################################### #
    #### 4. Input validation                               ####
    # ####################################################### #

    apollo_inputs = apollo_validateInputs()

    # ####################################################### #
    #### 5. Likelihood definition                          ####
    # ####################################################### #

    apollo_probabilities=function(apollo_beta, apollo_inputs,
                                  functionality="estimate"){

      ### Attach inputs and detach after function exit
      apollo_attach(apollo_beta, apollo_inputs)
      on.exit(apollo_detach(apollo_beta, apollo_inputs))

      ### Create list of probabilities P
      P = list()

      ### List of utilities: these must use the same names as
      ### in mnl_settings, order is irrelevant.
      V = list()
      V[['car']]  = asc_car  + b_tt*time_car  + b_c*cost_car
      V[['bus']]  = asc_bus  + b_tt*time_bus  + b_c*cost_bus
      V[['air']]  = asc_air  + b_tt*time_air  + b_c*cost_air
      V[['rail']] = asc_rail + b_tt*time_rail + b_c*cost_rail

      ### Define settings for MNL model component
      mnl_settings = list(
        alternatives  = c(car=1, bus=2, air=3, rail=4),
        avail         = list(car=av_car, bus=av_bus,
                             air=av_air, rail=av_rail),
        choiceVar     = choice,
        V             = V
      )

      ### Compute probabilities using MNL model
      P[['model']] = apollo_mnl(mnl_settings, functionality)

      ### Take product across observation for same individual
      P = apollo_panelProd(P, apollo_inputs, functionality)

      ### Average draws
      P = apollo_avgInterDraws(P, apollo_inputs, functionality)

      ### Prepare and return outputs of function
      P = apollo_prepareProb(P, apollo_inputs, functionality)
      return(P)
    }

    # ####################################################### #
    #### 6. Model estimation and reporting                 ####
    # ####################################################### #

    model = apollo_estimate(apollo_beta, apollo_fixed,
                            apollo_probabilities,
                            apollo_inputs,
                            list(writeIter=FALSE))

    apollo_modelOutput(model)

    #apollo_saveOutput(model)

    # ####################################################### #
    #### 7. Postprocessing of results                      ####
    # ####################################################### #

    ### Use the estimated model to make predictions
    predictions_base = apollo_prediction(model,
                                         apollo_probabilities,
                                         apollo_inputs)

    ### Now imagine the cost for rail increases by 10%
    ### and predict again
    database$cost_rail = 1.1*database$cost_rail
    apollo_inputs   = apollo_validateInputs()
    predictions_new = apollo_prediction(model,
                                        apollo_probabilities,
                                        apollo_inputs)

    ### Compare predictions
    change=(predictions_new-predictions_base)/predictions_base
    ### Not interested in chosen alternative now,
    ### so drop last column
    change=change[,-ncol(change)]
    ### Summary of changes (possible presence of NAs due to
    ### unavailable alternatives)
    summary(change)

## References

- Ben-Akiva, M. and Lerman, S. (1985) Discrete Choice Analysis. Cambridge, Massachusetts. The MIT Press. ISBN 978-0-262-02217-0
- Ben-Akiva, M.; McFadden, D.; Train, K.; Walker, J.; Bhat, C.; Bierlaire, M.; Bolduc, D.; Boersch-Supan, A.; Brownstone, D.; Bunch, D.; Daly, A.; De Palma, A.; Gopinath, D.; Karlstrom, A.; Munizaga, M. (2002) Hybrid Choice Models: Progress and Challenges. Marketing Letters 13, 163 - 175.
- Bhat, C. (2008) The multiple discrete-continuous extreme value (MDCEV) model: Role of utility function parameters, identification considerations,and model extensions. Transportation Research 42B, 274 - 303.
- Hancock, T.; Hess, S. and Choudhury, C. (2018) Decision field theory: Improvements to current methodology and comparisons with standard choice modelling techniques. Transportation Research 107B, 18-40.
- Pinjari, A. and Bhat, C. (2010) A multiple discrete–continuous nested extreme value (MDCNEV) model: Formulation and application to non-worker activity time-use and timing behavior on weekdays. Transportation Research 44B, 562 - 583.
- Train, K. (2009) Discrete Choice Methods with Simulation, 2nd edition. New York, New York. Cambridge University Press. ISBN 978-0-521-76655-5