[R] Allocate virtual memory on hard drive

[Prof Brian Ripley]

On 11/03/2013 16:45, Jie wrote:
> The vector contains 1.5*10^8 numeric elements. It takes about 3~4 GB in memory.
> And I would like to find percentiles: 0%, 0.5%, 1%, ... 100%
> I use 64 bit R and windows 7 with 24GB Ram.

So:

1) Try R 3.0.0 alpha.  Many operations on large vectors are more 
efficient there.

2) You could try --max-mem-size=32G, say.  In my experience Windows 
virtual memory management is too slow to be useful, but you could try ....

3) Add more RAM.  24GB is not a lot these days.

However, I tried this on a Linux box. Such a vector is only just over 
1GB and the maximum memory usage was 2.9GB.  Have you really told us the 
true story?

> Thank you.
>
> Best,
>
>
> On Mon, Mar 11, 2013 at 12:40 PM, jim holtman <jholtman at gmail.com> wrote:
>> R runs with data in memory.  What type of system are you running on (32 or
>> 64 bit)?  How big is your data; you did not provide much information about
>> your problem.  Depending on what you what to 'sort', there might be other
>> ways of doing it.  This gets back to my tag line: "Tell me what you want to
>> do, not how you want to do it".
>>
>> On Mon, Mar 11, 2013 at 11:20 AM, Jie <jimmycloud at gmail.com> wrote:
>>>
>>> Dear All,
>>>
>>> I have a long sequence and want to find the quantile, or sort it first.
>>> It seems sort() or quantile() reaches the memory limit.
>>> Is there a way to allocate more memoy on SSD for R when startup, so
>>> that R can use both RAM and hard drive space?
>>> Thank you.
>>>
>>> Best wishes,
>>> Jie
>>>
>>> ______________________________________________
>>> R-help at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>>
>>
>>
>> --
>> Jim Holtman
>> Data Munger Guru
>>
>> What is the problem that you are trying to solve?
>> Tell me what you want to do, not how you want to do it.
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>


-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595