[R] Memory allocation problem with large dataset

[Pamela Allen]

Hello All, 

I have a problem when I try and run an nlme model with an added correlation
structure on a large dataset.  This is not surprising, but I am not sure how
to fix this problem.  I am using R 2.6.1, and I have had similar problems in
S-plus.

My dataset is mass growth data from the same 8 animals over 10 years, and is
a data.frame with 23,598 rows and 5 columns, and using object.size(data)=
664448 bytes.  My first model, using the VonBertalanffy growth function:

vanfemfunc<-function(age.years, A, lk, t0)
A*((1-exp(-exp(lk)*(age.years-t0)))^3)

vanfemfit1<-nlme(model=mass~vanfemfunc(age.years, A, lk, t0), data=vanfem,
fixed=A+lk+t0~1, start=c("A"=201.8, "lk"=-1.17, "t0"=-2.71))

This fits fine, and has an object size of 3605660 bytes.  When I try this
second model that involves a correlation structure for the random effects:

vanfemfit2<-update(vanfemfit1, correlation=corAR1())

I receive the following error message:

Error: cannot allocate vector of size 392.9 Mb
In addition: Warning messages:
1: In corFactor.corAR1(object) :
  Reached total allocation of 1022Mb: see help(memory.size)

I have looked through all of the R-help archives, but nothing that was
previously suggested has worked so far.  I have a 32-bit Windows OS with 1
Gb RAM, and I have also tried all of the following functions on a 64-bit
Windows OS with 2 Gb RAM.  Here are the functions I have tried:

1.  Using --max-mem-size.  I changed the upper limit to 2047, which was the
maximum value allowed for both the 32-bit OS and the 64-bit OS. The error
message for vanfemfit2 now reads "Error: cannot allocate vector of size
392.9 Mb".

2.  Using the function memory.limit(size=4095). If I increase it any higher
for both the 32-bit and 62-bit OS, I get the error message "Don't be silly!:
your machine has a 4 Gb address limit".  
Now when I try the model, the error message says "Error: cannot allocate
vector of size 392.9 Mb" .  Calling memory.size(max=T) produces 1126.44 Mb
for the 32-bit OS, but 438.38 for the 64-bit OS.  
Calling memory.size(max=F) after I try vanfemfit2 produces 1038.02 Mb for
the 32-bit, and 417.90 for the 64-bit. 
If I use gc() and then call memory.size(max=F) again, the output is 240.90
for the 32-bit, and 13.69 for the 64-bit.  
If I use gc() and then try vanfemfit2 again, I get the same error message on
both computers.  

I cannot split up the data into smaller pieces to make this function work,
as I need all of the data to produce a proper growth curve, although I have
reduced the data to 2,000 to test vanfemfit2, and it did work.  Any advice
would be much appreciated.  Thank you!

-Pam Allen
MSc Candidate
allen at zoology.ubc.ca