[R-sig-ME] How can I estimate deviance explained of a mixed gamm?

[Ben Bolker]

On 14-09-27 01:35 PM, Jon Lopez wrote:
> Dear mixed modelers,

> I have already asked about this issue but never recived an answer
> back. So I will try again.  I have been modelling fish biomass
> according to some environmental parameters using mixed effect models
> (gamm4 package). I don't want to bore you with the details of my
> models since I believe that they are not significant to the point of
> this message.  However, please feel free to ask me about anything in
> case you think it is important. I have some GAMM candidates
> already. I am able to get AIC, BIC, R-sq, ... scores for these
> models but, unfortunately, I can't obtain deviance explained from
> them.

> I have found an interesting procedure to try to derive it, published by
> Gilman and colleagues in 2012. Here is the complete reference in case any
> of you want to take a look to it:
> 
> "Gilman, E., Chaloupka, M., Read, A., Dalzell, P., Holetschek, J., Curtice,
> C., 2012. Hawaii longline tuna fishery temporal trends in standardized
> catch rates and length distributions and effects on pelagic and seamount
> ecosystems. Aquatic Conservation: Marine and Freshwater Ecosystems 22(4),
> 446-488."
> 
> Nevertheless, the procedure explained in the paper above do not provide us
> with the exact score. Thus, I have been considering other options like
> using the deviance explained of a equivalent GAM with the random effect as
> a spline term [s(x, bs="re")] but I don't know how accurate it would be.

> Do you think both options can be used as an approximation for the
> GAMM's deviance explained? What are your feelings on that?

  The problem with determining "accuracy" is that we don't really
know what you're trying to measure when you say you want to quantify
"deviance explained".  The variety of solutions for computing measures
of goodness of fit for GLMs (Nagelkerke, Cox and Snell, etc.), for
LMMs, and for GLMMs suggests that the problem is more of defining
a sensible metric than computing it.  So can you be more precise
about what you want?

  I don't know.  *If* the deviances returned by gamm4 and lme4
are comparable (I don't know whether they are), then presumably
you just compute them both?

For reference, the Gilman et al. paper says:

There is no accepted way to formally estimate model fit for GAMMs
(Wood, 2006; Zuur et al., 2009). We developed and implemented an
approach by fitting an equivalent GAM to derive the percentage
deviance explained (a measure of GAM goodness-of-fit: see Hastie and
Tibshirani, 1990), and to evaluate the importance of explicitly
accounting for trip- and set-specific heterogeneity (the random
effects attributable to the sampling design constraints) using a
GAMM. This method had the following steps: (i) fit a GAM using the
same data and fixed effect variables as used in the GAMM and extract
the deviance residuals; (ii) fit a linear mixed effects model to the
residuals using a constant parameter only model with both trip and set
as the random effects; (iii) fit a linear fixed effects model to the
residuals using a constant parameter only model; and (iv) compare the
fit of the two linear models using Akaike Information Criterion (AIC)
and a log-likelihood ratio test (Wood, 2006). A smaller comparative
AIC value indicates a relatively better fitting model, and the formal
log-likelihood ratio test determines if the difference in deviance
between the GAMM (linear mixed effects regression) and GAM (linear
regression) models was significant. Hence, using both AIC as a guide
and the log-likelihood ratio test as a formal test we determined
whether inclusion of random effects was necessary. If the inclusion of
the random effects was found to be necessary, then we expect the GAMM
would account for more of the deviance than the equivalent GAM.