[R] manipulating large data and lmer

[Katherine Jones]

I have switched to R having lost use of SPSS. Unfortunately all my  
data was given to me by collaborators in SPSS files and the datasets  
are too big to put into excel and manipulate (100,000 records).

I am managing to import my data into R from SPSS with foreign, with  
no problems. I can do a hierarchical partitioning of variance  
analysis on the data and do basic stuff like calculate means.

I'm now trying to do mixed models with lmer (lme4 package). I can do  
the model, get the results (I realise it doesn't give p-values, not  
quite got my head round this yet, but am aware of authors post on why  
in R wiki; will need to read more about mixed models to fully  
understand his answer) and do some basic residuals plots. e.g. I can  
plot residuals versus fitted values.

My PROBLEM:-
-I can't plot residuals or fitted values against any of the  
variables. I think this is because the model and dataset are of  
different lengths due to lots of NA values. For my model, I specify  
na.exclude. Ideally I'd like to remove all the NA data at the  
read.table stage but I can't get it to do this. I also think I might  
be using the wrong code (am use lme examples in the R book by Crawley).
-I wish to extract fitted values for certain factors in my model e.g.  
sex. At the moment I can only get all of the fitted values, and not  
subset them.

I think perhaps I have dived in rather deep into stats given my  
sparse knowledge of R and of mixed models, but unfortunately I don't  
have use of SPSS or SAS to play with, so have no choice.

Any help is much welcomed. I am slowly overcoming my fear of no menus  
and starting to see the softwares potential.

Dr Katherine Jones
Department of Biology
Carleton University