[R-sig-ME] [Fwd: Re: Wald F tests]

[Ben Bolker]

  Doug's response makes perfect sense to me.

  However, from the on-the-ground, what-do-I-say-about-my-data-now
point of view, it seems that this is really an empirical question.
I would guess (wildly) that both the LRT and the Wald test would
converge asymptotically on the right answer.  **For classical ML
problems**, I have the feeling (unsupported by evidence!) that
LRT converges faster/is less wrong at any given value of N than
Wald tests (which, as you say, represent a second level of
approximation). I have no idea if this is true for GLMMs.
Really the only reason that I spoke against LRTs was that it
is well known (as shown e.g. in PB2000) that they are dicey for
LMMs, while the situation for Wald tests is relatively unknown.
In the absence of data, which is stronger: our prior belief that Wald
tests are bad because they're less reliable than LRT in some other
contexts, or our optimism that Wald tests aren't bad because they
haven't been shown to be so?

  If it really hasn't been done (and while I'm far from omniscient
I did *try* to review the literature on this topic, and have yet
to find an answer, or to have anyone on this list provide
an answer), I guess it's time to crank up
the old simulation engine and have a look ...

  For what it's worth, the results of the (possibly misguided)
inference survey so far are:

don't test hypotheses: 5
LRTs: 5
F/Wald tests: 7
bootstrap: 4
mcmcsamp: 8
randomization of null hyp: 5
AIC: 4
  + 2 write-ins:
  1 for "consilience of approaches"
  1 for BIC

  (out of 26 respondents)

  In hindsight, I would have liked to take mcmcsamp off the table
(or put it in a separate category) since I am really most interested in
finding out/telling researchers what to do NOW.

  cheers
    Ben Bolker

Douglas Bates wrote:
> On Tue, Oct 7, 2008 at 4:51 PM, Ben Bolker <bolker at ufl.edu> wrote:
> 
>>  But ... LRTs are non-recommended (anticonservative) for
>> comparing fixed effects of LMMs hence (presumably) for
>> GLMMs, unless sample size (# blocks/"residual" total sample
>> size) is large, no?
> 
>> I just got through telling readers of
>> a forthcoming TREE (Trends in Ecology and Evolution) article
>> that they should use Wald Z, chi^2, t, or F (depending on
>> whether testing a single or multiple parameters, and whether
>> there is overdispersion or not), in preference to LRTs,
>> for testing fixed effects ... ?  Or do you consider LRT
>> better than Wald in this case (in which case as far as
>> we know _nothing_ works very well for GLMMs, and I might
>> just start to cry ...)  Or perhaps I have to get busy
>> running some simulations ...
> 
> My reasoning, based on my experiences with nonlinear regression models
> and other nonlinear models, is that a test that involves fitting the
> alternative model and the null model then comparing the quality of the
> fit will give more realistic results than a test that only involves
> fitting the alternative model and using that fit to extrapolate to
> what the null model fit should be like.
> 
> We will always use approximations in statistics but as we get more
> powerful computing facilities some of the approximations that we
> needed to use in the past can be avoided.  I view Wald tests as an
> approximation to the quantity that we want to use to compare models,
> which is some measure of the comparative fit.  The likelihood ratio or
> the change in the deviance seems to be a reasonable way of comparing
> the fits of two nested models.  There may be problems with calibrating
> that quantity (i.e. converting it to a p-value) in which case we may
> want to use a bootstrap or some other simulation-based method like
> MCMC.  However, I don't think this difficulty would cause me to say
> that it is better to use an approximation to the model fit under the
> null hypothesis than to go ahead and fit it.
> 
>>  Where would _you_ go to find advice on inference
>> (as opposed to estimation) on estimated GLMM parameters?
> 
> I'm not sure.  As I once said to Martin, my research involves far too
> much "re" and far too little "search".  Probably because of laziness I
> tend to try to reason things out instead of conducting literature
> reviews.
> 
>>  cheers
>>   Ben Bolker
>>
>> Douglas Bates wrote:
>>> If I were using glmer to fit a generalized linear mixed model I would
>>> use likelihood ratio tests rather than Wald tests.  That is, I would
>>> fit a model including a particular term then fit it again without that
>>> term and calculate the difference in the deviance values, comparing
>>> that to a chi-square.
>>>
>>> I'm not sure how one would do this using the results from glmmPQL.
>>>
>>> On Fri, Oct 3, 2008 at 3:37 PM, Ben Bolker <bolker at ufl.edu> wrote:
>>>>  [forwarding to R-sig-mixed, where it is likely to get more
>>>> responses]
>>>>
>>>> Mark Fowler wrote:
>>>>
>>>> Hello,
>>>>        Might anyone know how to conduct Wald-type F-tests of the fixed
>>>> effects estimated by glmmPQL? I see this implemented in SAS (GLIMMIX),
>>>> and have seen it recommended in user group discussions, but haven't come
>>>> across any code to accomplish it. I understand the anova function treats
>>>> a glmmPQL fit as an lme fit, with the test assumptions based on maximum
>>>> likelihood, which is inappropriate for PQL. I'm using S-Plus 7. I also
>>>> have R 2.7 and S-Plus 8 if necessary.
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>