Frequently Asked Questions (FAQ)
Installation and updating of Apollo
How do I install Apollo?
Type install.packages("apollo") in the R console and press enter.
How do I update Apollo?
You update Apollo by re-installing it. Type install.packages("apollo") in the R console and press enter.
Why do some of my results differ from those obtained with an earlier version of Apollo?
Most results are unchanged across versions, but bug fixes and improvements can change some reported values, for example model fit statistics or standard errors, and changes to how models are estimated can make estimates differ in the last digits. All changes, and whether they affect results, are listed on the Version control page. We encourage users to always use the latest version of Apollo.
Why do I get an error during installation that one of the packages is not available for the R version I am running?
You are likely running an old version of R. Apollo, and the packages it uses, require a recent version of R, and the minimum version is listed on the Apollo page on CRAN. Update R to the latest version and re-install Apollo. You can get the latest version of R from https://cran.r-project.org/.
Why does installation work on my home laptop/desktop computer but fail on my company laptop/desktop computer?
Often computers from big organisations will install R packages in shared libraries (that is in folders in the private company network). R does not like its libraries to be in shared folders. As a general recommendation, always install packages in local libraries, i.e. in a folder on the local hard drive. You can see your active libraries by typing .libPaths(). This will list the active libraries. If the local library is, for example, the second one in the list, you should keep only that one by typing .libPaths(.libPaths()[2]). Then you should try installing Apollo again.
Data
Why do I get the error message Error in file(file, "rt") : cannot open the connection when trying to open the data?
This is most often caused by the working directory not being set to the folder containing the data file, meaning that R cannot find it, or by a typo in the name of the data file. The call to apollo_setWorkDir() at the start of a model file, as used in all the example files, sets the working directory to the folder containing the model file (cf. Chapter 2 of the manual), so that data files in that folder are found.
Can I use “long” formatted data in Apollo?
No. Apollo requires data to be in a “wide” format, meaning that all information necessary to calculate the likelihood of a single observation should be contained in a single row of the data. In more practical terms, for an MNL model, this means that attributes for all alternatives should be contained in each row. This format is the more common format in choice modelling, uses less space, and is also more general in allowing for a mixture of different dependent variables in the same data.
Is there a way to transform data in “long” format into “wide” format?
Yes, Apollo has a function for this called apollo_longToWide, described in Chapter 3 of the manual, though it may not work for every type of data.
Can I use a list of dummy variables to represent the choice? For example, for a choice between three alternatives, with the third chosen use variables alt1=0, alt2=0, alt3=1?
No. Apollo requires the user to encode the choice in a single variable. In this case, it would be a variable (for example called “choice”) that could take only three values (for example 1, 2 or 3). This is easily created in R on the basis of separate dummy variables.
Can I model “dual response” survey data with Apollo?
Yes, this is possible. We recommend beginning by modelling both questions separately, and if there is evidence of the parameters being similar, then estimate them jointly using a scale parameter between them. See BW_joint_model.r to learn how to conduct joint estimation in Apollo.
Can I use weights in Apollo?
Yes. The name of the variable containing the weights is set in apollo_control$weights, and apollo_weighting is called inside apollo_probabilities (cf. Section 4.5.3 of the manual). With panel data, apollo_weighting is usually called after apollo_panelProd, in which case the weights need to be the same for all observations of an individual. The weights are used in estimation and in other calculations of the likelihood, but the predictions returned by apollo_prediction are not weighted, and the number of observations used for BIC remains the number of modelled outcomes rather than the sum of the weights, which is reported separately in the output when the two differ.
What does the message about rows that are identical to another row mean?
When validating the data, Apollo checks for rows that are exact copies of other rows, and reports them in one of two ways. A message that some rows are “identical to another row apart from the individual’s ID” means that all other variables in these rows are the same, including any socio-demographic characteristics. This can happen legitimately when the data contains few or no characteristics of the respondents, for example when different respondents faced the same choice situation and made the same choice, but it can also point to respondents included more than once under different IDs, for example after merging files, and should then be checked. A warning that some rows are “exact copies of another row for the same individual” should be taken even more seriously, as this is often an error in the data, for example after merging or appending files. Neither message changes estimation.
Model specification
What distributions are possible for random coefficients in Apollo?
There are no limits imposed on distributional assumptions. The user can specify whatever distributions they want to use. Distributions can be coded as transformations of either Normal or Uniform draws. While transformations of Normal draws can be used for Lognormal, Censored or Truncated Normals and Johnson SB, Uniform distributions open up even broader scope as an inverse cumulative distribution function can be applied to the Uniform draws for a wide set of possible distributions.
How can I capture the panel structure of my data in Apollo?
The treatment of panel data depends completely on the model being used. Whenever the data contains multiple choices per individual, the analyst needs to use the function apollo_panelProd to group them together in estimation, except if using the setting apollo_control$panelData=FALSE, in which case the data will be treated as if all observations came from separate individuals. In models without any random heterogeneity, such as MNL, there is no explicit modelling of the correlation across choices for the same individual. All that will happen by using apollo_panelProd is that the calculation of the robust standard errors recognises that the choices come from the same individual. In models with random heterogeneity, such as Mixed Logit, the analyst can more explicitly account for the panel structure, for example by specifying that the heterogeneity in any random taste coefficients is across individuals, not within individuals, and/or by including an explicit pseudo-panel effect error component. These issues are discussed in detail in the manual.
How can I avoid writing each utility function separately if I have tens or hundreds of alternatives?
If the utilities all use the same structure but with different attributes for each alternative, then the utility functions (and availabilities) can be written iteratively. An example of this is provided in MNL_iterative_coding.r.
Can I use if statements inside apollo_probabilities?
Yes. Before estimation, Apollo rewrites the model functions (apollo_probabilities, apollo_randCoeff and apollo_lcPars) to make them faster, for example by expanding loops. An if whose condition depends only on settings, such as those in apollo_control, is replaced by the branch it takes. An if statement whose condition depends on parameters, data, draws, functionality or variables created inside the function is kept as written, as its condition can change from one call to the next. If such an if statement defines or changes an object (e.g. a utility), analytical gradients cannot be built, and Apollo uses numerical gradients instead, with a message saying so. Note that a condition on a parameter makes the likelihood discontinuous, which can cause problems for gradient-based estimation.
What starting values should I use for the thresholds in my Ordered Logit/Ordered Probit model?
The thresholds need to be different from each other, and monotonically increasing. If the thresholds are too wide, extreme ratings will obtain very low or zero probabilities. If the thresholds are too narrow, extreme ratings will obtain very large probabilities. Either of these can lead to estimation failures. Some trial and error is often required, but a good starting point is to have thresholds symmetrical around zero, going to extreme values of \(\pm3\).
How many draws should I use to estimate my models with random components?
There is no correct answer to this question. More draws is always better. The likelihood of the model is given by an integral without a closed form solution, and the simulation based approach only offers an approximation to this integral. Using a low number of draws means that the approximation to this integral is poor. In simple words, it means that the model we are estimating is not the one we think we are estimating. The parameter estimates will then be biased for the model we actually specified.
But the log-likelihood of my model is better with fewer draws, so isn’t that good?
The fact that the log-likelihood is better with fewer draws does not justify the use of fewer draws. It is simply a reflection of the fact that fewer draws offer a poor approximation to the real model. Once the number of draws is increased to a sufficient number, the model fit will stay much more stable for further increases.
Why does my model converge with a low number of draws, but fails with a high number of draws?
The fact that the model does not converge with a high number of draws shows that there is a problem with the model. It is known that using a low number of draws can mean a model that is overspecified still converges and can give every impression of being identified (Chiou and Walker, 2007).
Can I at least use fewer draws if I use quasi-Monte Carlo draws?
In theory, yes, but again, more is better. Care is also required in deciding which type of draws to use. Halton draws are an excellent option for models with a low number of random parameters, but the collinearity issues with Halton draws means they should not be used with more than say \(5\) random components.
To keep estimation cost under control, can I use a low number of draws in my specification search and then re-estimate the final model with a large number?
This is unfortunately rather common practice, but it is misguided to think this is a good approach or that it solves the issues arising with using low numbers of draws. The fact that low numbers are used during the specification search means that the approximation to the integral is poor at that stage. This in turn means that the decisions that lead to the final model specification may themselves be biased. While the final specification is then estimated robustly with a large number of draws, it may in fact be inferior to a specification that would have been obtained by using a high number throughout the specification search.
Does Apollo allow me to separate scale heterogeneity from preference heterogeneity?
The notion that it is possible to separate out scale heterogeneity from other heterogeneity is a myth, as discussed at length by Hess and Train (2017). Many models can allow for scale heterogeneity, but no model can separate it from preference heterogeneity, and there is no need to do so.
So can Apollo estimate the GMNL model?
The GMNL model is in fact not a new model or a more general model. It is simply a Mixed Logit model with a very particular set of constraints applied to it. It is not more general than Mixed Logit, which is the most general RUM model (cf. McFadden and Train, 2000). Given that Apollo allows full flexibility, users can of course specify the heterogeneity in a Mixed Logit model using GMNL-style constraints, but should be mindful when it comes to interpretation of the results given the above points.
So how about scale adjusted Latent Class (SALC)?
A SALC model is affected by the same issues as discussed by Hess and Train (2017) for GMNL. It is not possible to separate out scale heterogeneity from other heterogeneity. Users of Apollo can produce a SALC specification, which is simply a two layer Latent Class model, by using \(S_1\cdot S_2\) classes, allowing for \(S_1\) sets of \(\beta\) parameters and \(S_2\) sets of \(\mu\) (scale) parameters, with an appropriate normalisation, and with the \(S_1\cdot S_2\) classes using all combinations of \(\beta\) and \(\mu\). Such a model will be more general than a Latent Class model with \(S_1\) classes with different \(\beta\), but less general than a model with \(S_1\cdot S_2\) classes with different \(\beta\).
Errors and failures during estimation
Why does Apollo complain that some function arguments or settings are missing or incorrect?
Most Apollo functions take their inputs through a list of settings, such as mnl_settings for apollo_mnl or estimate_settings for apollo_estimate. Apollo checks the names of the elements in these lists, and stops with a message such as “SYNTAX ISSUE - The following entries in mnl_settings are not recognised: …”, followed by the list of recognised entries. This is most often caused by a typo in the name of a setting (R is case sensitive), or by a setting that belongs to a different model type, e.g. nlStructure in mnl_settings. Similarly, settings that are required for a model type, such as alternatives or choiceVar, need to be included. For functions that take several arguments, such as apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs), the arguments need to be given in the order shown in this manual. A different order is only possible if each argument is named in the call, e.g. apollo_estimate(apollo_inputs=apollo_inputs, apollo_beta=apollo_beta, apollo_fixed=apollo_fixed, apollo_probabilities=apollo_probabilities).
Why is my estimation failing with the message “Log-likelihood calculation fails at starting values”?
This happens when, at the starting values, the likelihood of the model is zero or cannot be calculated for at least some people in the data. Apollo will report the IDs of these individuals. A related message, “Log-likelihood calculation fails at values close to the starting values”, means that the likelihood can be calculated at the starting values, but not after Apollo adds small disturbances to them to test whether each parameter influences the likelihood. This shows that the starting values are at, or very close to, the edge of the region where the model is defined, and the same remedies apply. Three common reasons exist for these problems:
Are the starting values feasible/appropriate for the model?
The most common reason for the initial likelihood calculation to fail is a problem with the values used in apollo_beta. The starting values of some parameters may be invalid. For example, the model may be dividing by a parameter with an initial value equal to zero. Also, different models have different requirements, for example the structural parameters in nested and Cross-nested Logit models should be different from zero; the \(\alpha\) parameters in Cross-nested Logit models should be between zero and one; the \(\gamma\) and \(\sigma\) parameters in MDCEV should be greater than zero; the thresholds in Ordered Logit and Probit models should be different and monotonically increasing; the variance of linear regressions should be positive; etc. Another potential cause (less common than the previous one) is that starting values are too poor, leading to an initial likelihood too close to zero, which in turn leads to an infinite log-likelihood. To avoid this, the user should look for better starting values, either by estimating a simpler model and using those estimates as starting values, or using the apollo_searchStart function.
Does the data contain many observations for each person and/or does the model use several components (e.g. hybrid choice)?
When multiplying together the probability of many individual observations at the person level (using apollo_panelProd), it is possible that the product becomes too close to zero for R to store it as a number. The same can happen when combining many individual model components using apollo_combineModels. The risk of this is greater in case of models with low probabilities, such as in the case of large choice sets or many observations per individual. A solution to this problem is to set workInLogs=TRUE in apollo_control. This ensures that all calculations are made with the logarithms of probabilities, avoiding the issue of multiplying many small numbers. This setting only applies to panel data, and in models with several components, apollo_panelProd then needs to be called only once, after apollo_combineModels, rather than after each model component as in Section 7.3 of the manual. The use of this setting is however only recommended when necessary as it will slow down estimation and also prevents the use of analytical gradients.
Does the model use lognormal distributions for random coefficients?
The value of a lognormally distributed coefficient is given by \(\beta=\exp\left(\mu_{\ln\left(\beta\right)}+\sigma_{\ln\left(\beta\right)}\xi\right)\), or \(\beta=-\exp\left(\mu_{\ln\left(-\beta\right)}+\sigma_{\ln\left(-\beta\right)}\xi\right)\) in the case of a negative lognormal distribution. The estimated parameters thus relate to the mean and standard deviation of the logarithm of \(\beta\) (or the logarithm of \(-\beta\)). A common mistake is to start \(\mu_{\ln\left(\beta\right)}\) at zero, just as in the case of a normally distributed \(\beta\). With the exponential, this will lead to a large starting value for \(\beta\), which can result in numerical problems. In the case of lognormally distributed coefficients, it is thus advisable to use a large negative value for the starting value of the mean of the logarithm of the coefficient, e.g. starting \(\mu_{\ln\left(\beta\right)}\) at something like \(-3\) or lower, as this would imply starting the median of \(\beta\) close to zero.
Why do I get an error saying that one of my parameters does not influence the likelihood, even though I am using it in a utility function?
There may be several reasons for this. A common mistake is that, when writing utilities (or any other code statement) across multiple lines, the link between lines is missing and only the first one is considered. To split a statement across multiple lines, the incomplete lines should finish with an operator. For example:
U[["A"]] = b0 + b1*x1
+ b2*x2will ignore the effect of b2*x2. Instead, it should be:
U[["A"]] = b0 + b1*x1 +
b2*x2or
U[["A"]] = (b0
+ b1*x1
+ b2*x2)It could also be that the attribute associated with the parameter does not vary across the utility functions, or that the same constant is included in all utility functions. Much less common, it could be that the starting probabilities in your model are so small that due to rounding errors, they are equal to zero, and changes in parameter values also lead to small probabilities not different from zero. This is more likely to happen in complex models with many observations per individual. In this case, we recommend (1) to begin by estimating a simple model (e.g. constants only) to obtain better starting values, and (2) to set the option apollo_control$workInLogs=TRUE. This last option will increase numerical precision at the expense of estimation speed, and also prevents the use of analytical gradients.
Estimation of my model failed after a long time. Have I lost all the information?
No. With the bgw and bfgs estimation routines, Apollo saves the parameter values at each iteration in a csv file called modelName_iterations.csv in the output directory (cf. outputDirectory in apollo_control), where modelName is as defined in apollo_control, unless estimate_settings$writeIter=FALSE. The last row of this file can be used as starting values for a new estimation.
What happens if the maximum number of iterations is reached?
The default number of iterations for estimation is set to \(200\). If this limit is reached, Apollo returns the estimates from the last iteration with a warning that the iteration limit was exceeded, and does not compute the covariance matrix, unless estimate_settings$forceCovarianceMatrix=TRUE. The estimates can then be used as starting values for a new estimation, and the limit can be increased in estimate_settings$maxIterations. In general however, if a model has not converged after \(200\) iterations, this could be a sign of problems with the model. Inspecting the iterations file produced during estimation can help diagnose if there is a problem or if more iterations are indeed required.
Why does Apollo say that it will use numerical gradients, and does it matter?
Apollo uses analytical gradients by default for all model components, which makes estimation, and the calculation of the covariance matrix, considerably faster. In some cases these cannot be built, and Apollo then uses numerical gradients, with a message saying so, for example with workInLogs=TRUE, when an if statement inside the model functions defines or changes an object, or when a utility calls a function defined by the user that Apollo cannot expand (see Section 4.6 of the manual for the full list). The model is still estimated correctly, but more slowly. The analytical gradients can be compared with numerical ones before estimation using estimate_settings$validateGrad=TRUE, and switched off with apollo_control$analyticGrad=FALSE.
How many cores should I use for estimation?
This depends on the model, the data and the computer. Using more cores (nCores in apollo_control) reduces estimation time, but with diminishing returns, and the data are copied to each core, which increases memory use. If R uses more memory than the physical memory available, estimation slows down considerably. The function apollo_speedTest (cf. Section 6.5 of the manual) measures the evaluation time for different numbers of cores and draws, and can help with this decision. If the computer is used for other tasks during estimation, no more than the number of cores in the machine minus one should be used. Functions that estimate a model many times, such as apollo_bootstrap and apollo_outOfSample, start a new set of workers for each repetition, so for models that estimate in a few seconds, nCores=1 may be faster.
Model results
Why am I getting Inf or NaN for standard errors?
There are several reasons why this could happen:
The model could have theoretical identification issues
To diagnose these issues is not easy, as requirements change depending on the particular structure of the model. For example, in random utility models, only differences in utility matter, so the constant of at least one alternative must be fixed to zero. Similarly, a normalisation is required for categorical variables. In Hybrid choice models, the variance of each structural equation (or the slope of one measurement equation per latent variable) should be fixed to one.
The model could be too complex for the data, leading to empirical identification issues
Many users fall into the trap of believing that choice models are easy tools and that the most complex model should always be used. Instead, analysts should always begin by estimating the simplest possible model and moving progressively towards more complex formulations. This will help troubleshooting any potential identification problems. In Mixed Logit for example, analysts should always start by introducing only a few random coefficients, leaving the rest as fixed, and progressively making the model more general.
Differences in scale between parameters can complicate the calculation of the standard errors
Calculating standard errors requires inverting the Hessian matrix at the estimated value of the parameters. Depending on the model, this Hessian is calculated using numerical derivatives of the analytical gradient or of the log-likelihood, looking at small changes to either side of the estimates. Some of these could result in numerical failures. Similarly, inverting the Hessian itself can be challenging due to numerical precision issues if the parameters are on very different scales. This can be diagnosed by looking at the diagonal of model$hessian: if its elements differ by several orders of magnitude, the parameters are on very different scales. More generally, the eigenvalues in model$hessianEigenValue show whether the Hessian is close to singular: a very large ratio between the largest and smallest eigenvalue (in absolute value) means that inverting it is problematic, whether because of differences in scale, which the options below address, or because of identification issues, which they do not. In these cases, the simplest option is to set estimate_settings$scaleHessian=TRUE, which rescales the parameters by their estimates before calculating the Hessian, without changing the estimation. Alternatively, the estimate_settings$scaling option can help. This is an optional named vector that can be given to the apollo_estimate function to scale individual parameters, e.g. scaling=c(b_cost=0.01) for a parameter whose estimate is around \(0.01\), and therefore avoid numerical issues (cf. Section 4.6 of the manual). A different routine for computing the Hessian can also be tried, using estimate_settings$hessianRoutine.
The calculation of numerical derivatives could lead to some zero probabilities
Especially with complex models, the calculation of the numerical derivatives can be affected by a small number of calculations leading to zero probabilities. Greater stability can in this case be obtained by using bootstrapping for estimating the standard errors (i.e. setting estimate_settings$bootstrapSE to the number of bootstrap samples, e.g. bootstrapSE=100), obviously at the cost of increased estimation time. Similarly, depending on the data used, setting workInLogs=TRUE in apollo_control could help, but at the expense of estimation speed, and it also prevents the use of analytical gradients.
What can I do if the covariance matrix cannot be computed at all?
If the Hessian cannot be computed or inverted with one routine, Apollo tries the next one, from "analytic" to "numDeriv" and then "maxLik" (cf. hessianRoutine in Section 4.6 of the manual). The BHHH covariance matrix, which only uses first derivatives, is also returned in model$BHHHvarcov, and the BHHH standard errors can be printed using printBHHH=TRUE in apollo_modelOutput. These can help find the parameters that cause the problem, but are not a substitute for the classical and robust standard errors, as a Hessian that cannot be computed or inverted is usually a sign of the identification issues discussed above.
Why is my estimate of the standard deviation negative?
This happens if a random coefficient is coded as randCoeff[["beta"]]=mu+sigma*draws where draws is a random variate that is symmetrical around zero.
So should I constrain the standard deviation to be positive?
There is no reason for doing so. The results will be the same if the random variate draws is symmetrical around zero. Imposing constraints will also make estimation harder. And of course, if a user wants to allow for correlation between individual coefficients, then the parameters multiplying the draws need to be able to be positive or negative.
Why are my structural/nesting parameters greater than one or below zero in Nested or Cross-nested Logit?
Apollo does not automatically constrain the structural parameters in Nested (NL) and Cross-nested (CNL) Logit models to be between zero and one. If, after estimation, the structural parameters are outside of this interval, this could be evidence of the nesting structure not being supported by the data. The user should then try a different nesting structure.
So should I constrain them to be between \(0\) and \(1\)?
In general, we do not recommend imposing constraints. If unconstrained estimation yields an estimate outside the bounds of the interval of acceptable values, then it is highly likely that the use of constraints will lead to an estimate that goes to one of the bounds of the interval that the parameter is constrained in. The model fit will be inferior too and the real problem will simply be masked by the constraints. If constraints are needed, they are set using estimate_settings$constraints (cf. Section 4.6 of the manual). Equality constraints are substituted into the model, which keeps the estimation routine and analytical gradients, while inequality constraints require the bfgs routine, which is generally slower than the default bgw.
So how about constraints on other parameters, such as standard deviations or \(\gamma\) parameters in MDCEV?
It is common practice to use exponential transforms for parameters that are only allowed to be positive, e.g. using \(\gamma=\exp\left(\gamma_0\right)\), with \(\gamma_0\) being estimated. We have found that this often slows down estimation and does not necessarily lead to the same solution as unconstrained estimation even if the latter finds an acceptable solution. The reason for the problem is that small changes to \(\gamma_0\) will lead to large changes in \(\gamma\), making estimation difficult, especially with numerical derivatives.
How do I know whether the chains of my HB model have converged?
Apollo reports convergence statistics for the parameter chains based on the Geweke test (cf. Section 8.1 of the manual), which apollo_saveOutput includes by default, and values above \(1.96\) in absolute value indicate that a chain has not converged. Apollo also warns if, at the end of the burn-in, the acceptance rate of the Metropolis-Hastings steps is still far from its target. If the chains have not converged, the number of burn-in iterations (gNCREP) and of iterations kept after the burn-in (gNEREP) should be increased, and the specification should be checked for parameters that are only weakly identified. Running the same model with different values of gSeed and comparing the results is a further check. Just as results obtained with too few draws in classical estimation, results from chains that have not converged should not be used.
How do I calculate willingness-to-pay measures and their standard errors?
For coefficients that do not vary randomly, the willingness-to-pay for an attribute is the ratio of its coefficient to the cost coefficient, and the standard error of this ratio can be computed with the Delta method using apollo_deltaMethod (cf. Section 9.13 of the manual), e.g. with deltaMethod_settings=list(expression=c(wtp_time="b_time/b_cost")). Several expressions can be given at once, and they can be any function of the parameters. With random coefficients, the willingness-to-pay is itself random, and its distribution can be obtained from apollo_unconditionals, or at the individual level from apollo_conditionals (cf. Section 9.15.1 of the manual). Alternatively, the model can be specified directly in willingness-to-pay space, as in MMNL_wtp_space_inter_intra.r. The function apollo_deltaMethod cannot be used for models estimated with HB.
How do I calculate hit rates for my model in Apollo?
We made the decision not to include hit rates in Apollo outputs. They really offer a very distorted view of results. Models give probabilities in prediction. If, for each task, an analyst just looks at what alternative has the highest probability, then they are ignoring the error term in the model. To put it succinctly, imagine you have a case with 2 alternatives, and we have 2 models. Model A gives a probability of 51% for the chosen alternative in 70% of cases, but a probability of only 10% in the remaining 30% of cases. Model B gives a probability of 49% for the chosen alternative in 70% of cases, but a probability of 90% in the remaining 30% of cases. Using a hit rate would give model A a figure of 70%, and model B a figure of 30%. But clearly model B is far superior. This is why choice modellers work with probabilities of correct prediction instead if they want a percent measure like this. That would give 0.387 for model A but 0.613 for model B.