[R] lm Regression takes 24+ GB RAM - Error message

[Milan Bouchet-Valat]

Le mercredi 06 mars 2013 à 09:18 -0800, Jonas125 a écrit :
> Length(Datasplit) = 7100
> 
> I did a regression for Datasplit[[1]] and the calculated columns --> the
> object size is 70 MB. Quite large....
7100*70/1024 = 485 (GB)

No wonder why you run out of memory quite fast.

You probably do not need to store the whole lm objects: usually you need
coefficients, R-squared, things like that. So instead of returning the
objects, return a vector or a list with only the elements you need, you
will save much space.

And if you really need the objects, set these lm() arguments to FALSE to
make the result smaller:
model, x, y, qr: logicals.  If ‘TRUE’ the corresponding components of
          the fit (the model frame, the model matrix, the response, the
          QR decomposition) are returned.

> Assuming that R cannot handle inf values in regressions (didn't have the
> time to google it)
> How can I avoid the calculation of infinite values? Like "If the denominator
> would be zero, choose 0.0000001 as the denominator instead." 
> Dataset[is.infinite(Dataset)] <- 0 does not work for me --> "default method
> not implemented for type 'list' "
> class(Dataset) = data.frame
I don't understand why you think infinite values can trigger a memory
problem. Why don't you just try it?

> lm(c(1, Inf) ~ c(1, 2))
Erreur dans lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) : 
  NA/NaN/Inf in 'y'
> lm(c(1, 2) ~ c(1, Inf))
Erreur dans lm.fit(x, y, offset = offset, singular.ok = singular.ok, ...) : 
  NA/NaN/Inf in 'x'

So, if anything, this would stop your lapply() call sooner or later, and
save your machine from freezing.



Regards


> 
> 
> --
> View this message in context: http://r.789695.n4.nabble.com/lm-Regression-takes-24-GB-RAM-Error-message-tp4660434p4660501.html
> Sent from the R help mailing list archive at Nabble.com.
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.