[R-sig-ME] Fwd: Re: Wald F tests

[Ben Bolker]

  Yes, this is partly tongue in cheek, and I agree that
hypothesis testing is overemphasized (I suspect that many
of the r-sig-mixed-models regulars would also agree).  Let's
say we want to construct confidence intervals rather than
test null hypotheses.  Then our choices are something like

* construct Z- or t-based confidence intervals from
estimated standard error
* bootstrap confidence intervals
* mcmcsamp confidence intervals

  which correspond to c,d,e below.  I suppose another
choice (corresponding more or less to b, LRT)
would be likelihood profile
confidence intervals, but I would really worry in
this case that the known anticonservatism of LRTs
would translate to profile confidence intervals
with poor coverage.

Most of the difficulties that arise in null-hypothesis testing have
analogues in constructing appropriate confidence intervals.

  cheers
    Ben Bolker

>> c F tests (LMM) or Wald tests (GLMM) [ignore mismatch with hypothesized
>> null distributions]
>> d bootstrapped confidence intervals
>> e [mcmcsamp confidence intervals -- if available]
>> f randomization/simulation tests of nested null hypotheses
>> g AIC comparisons [ignore that prediction != hypothesis testing]

John Maindonald wrote:
> I think it is going too far to say that one should not be
> testing hypotheses (the implication of that "is soooo 20th C"?).
> But the place of that activity is much more limited than is
> commonly recognized.
> 
> Basically, I do not like the range of options that this (half-serious?)
> survey has on offer, and I'd need to write half a page or more
> to explain why.  Democracy maybe, but (as I suppose is
> always the case in the political democracies that are on offer)
> the choices are severely constrained.
> 
> Where such a hypothesis testing perspective may be
> appropriate, the preferred starting point is almost always
> a confidence interval.  Why not ask the comparable questions
> arise for estimation?
> 
> There's an editorial in Volume 72(5) (pp.1057-1058) of the
> Journal of Wildlife Management with which I pretty much agree:
> "... understand that the average reader of the Journal is
> interested in the biological questions addressed with your
> work.  The analytical framework and resulting results should
> support those questions and flow from them, not overwhelm
> them."
> 
> But I guess that Ben would like us to assume that the proper
> support framework is in place!
> 
> Note also, on pages 1272-1278 of the same issue:
> "Suggestions for Basic Graph Use When Reporting Wildlife
> Research Results", by Brett Collier.
> 
> John Maindonald             email: john.maindonald at anu.edu.au
> phone : +61 2 (6125)3473    fax  : +61 2(6125)5549
> Centre for Mathematics & Its Applications, Room 1194,
> John Dedman Mathematical Sciences Building (Building 27)
> Australian National University, Canberra ACT 0200.
> 
> 
> On 11/10/2008, at 6:47 AM, Ben Bolker wrote:
> 
>>
>> Does anyone out there have opinions on this subject?
>> How should one test hypotheses about fixed effects in
>> (G)LMMs, especially for small to moderate sample sizes?
>> (Please ignore issues of _estimation_ (PQL
>> vs Laplace vs AGQ vs ...)
>>
>> Should it amuse you to do so, you can vote at:
>>
>> http://www.surveymonkey.com/s.aspx?sm=yLyfrV_2ftw6WGx2dEFLWnIw_3d_3d
>>
>> (since we all know that scientific questions are settled
>> by a democratic process)
>>
>> a hypothesis testing is soooo 20th century, don't bother
>> b likelihood ratio tests [ignore known anticonservatism]
>> c F tests (LMM) or Wald tests (GLMM) [ignore mismatch with hypothesized
>> null distributions]
>> d bootstrapped confidence intervals
>> e [mcmcsamp confidence intervals -- if available]
>> f randomization/simulation tests of nested null hypotheses
>> g AIC comparisons [ignore that prediction != hypothesis testing]
>>
>> Note that Wald Z tests [option c] are more or less what you're
>> doing, implicitly, if you just eyeball the estimated
>> parameter values and their standard errors.
>>
>> cheers
>>   Ben Bolker
>>
>> -------- Original Message --------
>> Subject: [Fwd: Re: [R-sig-ME] Wald F tests]
>> Date: Tue, 07 Oct 2008 17:51:01 -0400
>> From: Ben Bolker <bolker at ufl.edu>
>> To: R Mixed Models <r-sig-mixed-models at r-project.org>
>>
>>
>> But ... LRTs are non-recommended (anticonservative) for
>> comparing fixed effects of LMMs hence (presumably) for
>> GLMMs, unless sample size (# blocks/"residual" total sample
>> size) is large, no?
>>
>> I just got through telling readers of
>> a forthcoming TREE (Trends in Ecology and Evolution) article
>> that they should use Wald Z, chi^2, t, or F (depending on
>> whether testing a single or multiple parameters, and whether
>> there is overdispersion or not), in preference to LRTs,
>> for testing fixed effects ... ?  Or do you consider LRT
>> better than Wald in this case (in which case as far as
>> we know _nothing_ works very well for GLMMs, and I might
>> just start to cry ...)  Or perhaps I have to get busy
>> running some simulations ...
>>
>> Where would _you_ go to find advice on inference
>> (as opposed to estimation) on estimated GLMM parameters?
>>
>> cheers
>>  Ben Bolker
>>
>> Douglas Bates wrote:
>>> If I were using glmer to fit a generalized linear mixed model I would
>>> use likelihood ratio tests rather than Wald tests.  That is, I would
>>> fit a model including a particular term then fit it again without that
>>> term and calculate the difference in the deviance values, comparing
>>> that to a chi-square.
>>>
>>> I'm not sure how one would do this using the results from glmmPQL.
>>>
>>> On Fri, Oct 3, 2008 at 3:37 PM, Ben Bolker <bolker at ufl.edu> wrote:
>>>> [forwarding to R-sig-mixed, where it is likely to get more
>>>> responses]
>>>>
>>>> Mark Fowler wrote:
>>>>
>>>> Hello,
>>>>      Might anyone know how to conduct Wald-type F-tests of the fixed
>>>> effects estimated by glmmPQL? I see this implemented in SAS (GLIMMIX),
>>>> and have seen it recommended in user group discussions, but haven't
>>>> come
>>>> across any code to accomplish it. I understand the anova function
>>>> treats
>>>> a glmmPQL fit as an lme fit, with the test assumptions based on maximum
>>>> likelihood, which is inappropriate for PQL. I'm using S-Plus 7. I also
>>>> have R 2.7 and S-Plus 8 if necessary.
>>>>
>>>> _______________________________________________
>>>> R-sig-mixed-models at r-project.org mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>>>
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models