[R] R dynamic memory management

[Maura E Monville]

I forgot to attach my script. I work under pressure and make a lot of
these mistakes. Sorry.
I have attached it right now. I'm lso pasting the messages appearing
on R GUI in the following.
I see the average becomes NA which makes me think that for some reason
R interpreter is reading out of the loaded file ...
I will try to locate the file that causes this problem.
Thanks to all of you.

Maura

start 4563 end 161 avg -0.04171546
start 4724 end 4871 avg -2.880717
gc():  242445 107937 6.5 0.9 467875 786432 12.5 6 350000 786432 9.4 6
Read 1 item
i from which: 7
start 1 end 135 avg -2.993604
start 136 end 135 avg -1.807748
start 271 end 176 avg -3.047341
start 447 end 154 avg -2.036649
start 601 end 141 avg -1.0865
start 742 end 152 avg -0.6664973
start 894 end 121 avg -0.6277448
start 1015 end 160 avg -0.4067699
start 1175 end 161 avg -0.4687507
start 1336 end 185 avg -0.478178
start 1521 end 211 avg -0.5239797
start 1732 end 600 avg -1.581398
start 2332 end 288 avg -0.5022956
start 2620 end 144 avg -0.07848339
start 2764 end 215 avg -0.2710712
start 2979 end 178 avg -0.1480546
start 3157 end 205 avg -0.2299476
start 3362 end 180 avg -0.1381425
start 3542 end 141 avg -0.1376570
start 3683 end 151 avg -0.111202
start 3834 end 134 avg -0.1246371
start 3968 end 142 avg -0.09887487
start 4110 end 126 avg -0.1023989
start 4236 end 118 avg -0.08591205
start 4354 end 154 avg -0.08125074
start 4508 end 187 avg -0.1366167
start 4695 end 178 avg -0.1046542
start 4873 end 178 avg -0.1076104
start 5051 end 2 avg -0.08889775
start 5053 end 4 avg NA
start 5057 end 8 avg NA
start 5065 end 16 avg NA
start 5081 end 32 avg NA
start 5113 end 64 avg NA
start 5177 end 128 avg NA
start 5305 end 256 avg NA
start 5561 end 512 avg NA
start 6073 end 1024 avg NA
start 7097 end 2048 avg NA
start 9145 end 4096 avg NA
start 13241 end 8192 avg NA
start 21433 end 16384 avg NA
start 37817 end 32768 avg NA
start 70585 end 65536 avg NA
start 136121 end 131072 avg NA
start 267193 end 262144 avg NA
start 529337 end 524288 avg NA
start 1053625 end 1048576 avg NA
start 2102201 end 2097152 avg NA
start 4199353 end 4194304 avg NA
start 8393657 end 8388608 avg NA
start 16782265 end 16777216 avg NA
Error: cannot allocate vector of size 32.0 Mb
In addition: Warning messages:
1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
>



On 10/9/07, jim holtman <jholtman at gmail.com> wrote:
> Have you tried using 'gc()' at the end of your loop each time you free
> up the objects that have been use 'rm(.....)'?  You might also look at
> the output from gc() to see if memory is being used up.  You may be
> getting fragmentation in memory.
>
> If you are processing a list of files and this continues, then one way
> is to process a subset, then exit and restart R and process the next
> subset.  It would be useful to know the sizes of the objects that you
> are processing and how you are writing them out.  Are you using 'save'
> or 'write.table' ('save' would be a lot more efficient).  Since we
> have no idea of what you loop looks like it is hard to provide more
> suggestions.
>
> On 10/9/07, Maura E Monville <maura.monville at gmail.com> wrote:
> > I have a loop that processes all the raw data files in agiven directory.
> > It should allocate only the memory to accomodate the filenames list.
> > Each file is loaded into RAM one at a time and saved to another
> > directory after processing. The R script runs fine but it had some
> > memory problems so it was terminated before getting to the end of the
> > list.
> > This is a Windows XP desktop with ~ 1GB RAM
> > In the following I am pasting R messages. I would appreciate some
> > suggestion about memory handling:
> >
> > Error: cannot allocate vector of size 32.0 Mb
> > In addition: Warning messages:
> > 1: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
> > 2: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
> > 3: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
> > 4: Reached total allocation of 958Mb: see help(memory.size) in: x[[i]] <- y
> >
> >
> > --
> > Maura E.M
> >
> > ______________________________________________
> > R-help at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
> >
>
>
> --
> Jim Holtman
> Cincinnati, OH
> +1 513 646 9390
>
> What is the problem you are trying to solve?
>


-- 
Maura E.M