[R] ERROR : cannot allocate vector of size (in MB & GB)

[arun]

HI,

You can try like using dbLoad() from hash package to load.  Also, if you need to chunk the data, you can use ff package.

A.K.





----- Original Message -----
From: Rantony <antony.akkara at ge.com>
To: r-help at r-project.org
Cc: 
Sent: Tuesday, July 24, 2012 9:45 AM
Subject: [R] ERROR : cannot allocate vector of size (in MB & GB)

Hi,

Here in R, I need to load a huge file(.csv) , its size is 200MB. [may come
more than 1GB sometimes].
When i tried to load into a variable it taking too much of time and after
that when i do cbind by groups,
getting an error like this

" Error: cannot allocate vector of size 82.4 Mb "

My requirement is, spilt data from Huge-size-file(.csv) to no. of small csv
files.
Here i will give no of lines to be 'split by' as input.

Below i give my code
-------------------------------
        SplitLargeCSVToMany <- function(DataMatrix,Destination,NoOfLineToGroup)
        {
            test <- data.frame(read.csv(DataMatrix))
            
            # create groups No.of rows 
            group <- rep(1:NROW(test), each=NoOfLineToGroup) 
            new.test <- cbind(test, group=group) 
            new.test2 <- new.test
            new.test2[,ncol(new.test2)] <- NULL
            
            # now get indices to write out 
            indices <- split(seq(nrow(test)), new.test[, 'group'])
            
            # now write out the files 
            for (i in names(indices))
            { 
            write.csv(new.test2[indices[[i]],], file=paste(Destination,"data.", i,
".csv", sep=""),row.names=FALSE) 
            }             
        }

-----------------------------------------------------
My system Configuration is,
Intel Core2 Duo
speed : 3GHz
2 GB RAM
OS: Windows-XP [ServicePack-3]
---------------------------------------------------

Any hope to solve this issue ?

Thanks in advance,
Antony.




--
View this message in context: http://r.789695.n4.nabble.com/ERROR-cannot-allocate-vector-of-size-in-MB-GB-tp4637597.html
Sent from the R help mailing list archive at Nabble.com.

______________________________________________
R-help at r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.