[R] lm Regression takes 24+ GB RAM - Error message

[R. Michael Weylandt]

On Wed, Mar 6, 2013 at 9:51 AM, Jonas125 <schleeberger.j at pg.com> wrote:
> Hello,
>
> I am a rather unexperienced r-user (learned the language 1 month ago) and
> run into the following problem using a local computer with 6 cores & 24 GB
> RAM and R 2.15 64-bit. I didn't install any additional packages
>
> 1. Via the read.table command I load a data table (with different data
> types) which is about 730 MB large
> 2. I add 2 calculated columns
> 3. I split the dataset by 5 criteria
> 4. I run the lm command on the split with the calculated columns as the
> variables
>
> The RAM consumption goes rapidly up and stays at 24 GB for a couple of
> minutes.
> The result:
> Error: cannot allocate vector size of 5.0 Mb
> In addition: There ware 50 or more warnings (use warnings() to see the first
> 50)
> --> Reached total allocation of 24559Mb

So it seems R has access to all your memory.

My guess is that you have so-called "factors" [Categorical variables]
in your dataset and this makes the linear regression a much larger
calculation (in the intermediate steps) than you might realize because
the design matrix has to deal with all the crossed categories.

Can you provide the output of str(DATA_SET)?

MW

>
> My code works perfectly fine for a smaller dataset. I am surprised about the
> errors as the CPU should do all the work with the lm calculations and the
> output cannot be that large, can it??? (I cannot check the object size of
> the lm object due to the error)
>
> Right now I am running only 1 linear model, but actually I wanted to run 6!
>
> Is Windows putting some restrictions on R regarding the RAM usage? Can I
> change any settings?
> A RAM upgrade is not an option. Do I need to use a different R package
> instead (bigmemory?)?

Not a bad idea.

>
>
> Thanks in advance for your help!!
>
>
>
>
>
> --
> View this message in context: http://r.789695.n4.nabble.com/lm-Regression-takes-24-GB-RAM-Error-message-tp4660434.html
> Sent from the R help mailing list archive at Nabble.com.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.