[R] how to parallelize 'apply' across multiple cores on a Mac

[David Romano]

(I neglected to use reply-all.)

---------- Forwarded message ----------
From: David Romano <dromano at stanford.edu>
Date: Sat, May 4, 2013 at 11:25 AM
Subject: Re: [R] how to parallelize 'apply' across multiple cores on a Mac
To: Charles Berry <ccberry at ucsd.edu>


On Sat, May 4, 2013 at 9:32 AM, Charles Berry <ccberry at ucsd.edu> wrote:
> David,
>
> If you insist on explicitly parallelizing this:
>
> The functions in the recommended package 'parallel' work on a Mac.
>
> I would not try to work on each tiny column as a separate function call -
> too much overhead if you parallelize - instead, bundle up 100-1000 columns
> to operate on.
>
> The calc's you describe are sound simple enough that I would just write
> them in C and use the .Call interface to invoke them. You only need enough
> working memory in C to operate on one column and space to save the result.
>
> So a MacBook with 8GB of memory will handle it with room to breathe.
>
> This is a good use case for the 'inline' package, especially if you are
> unfamiliar with the use of .Call.
>
>
> ===
>
> But it might be as fast to forget about paralleizing this (explicitly).
>
[detailed recommendations deleted]
>
> On a Mac, the vecLib BLAS will do crossprod using the multiple
> cores without your needing to do anything special. So you can forget about
> 'parallel', 'multicore', etc.
>
>
> So your remaining problem is to reread steps 2=6 and figure out what
> 'minimal.matrix' and 'fill.rows' have to be.
>
> ===
>
> You can also approach this problem using 'filter', but that can get
> 'convoluted' (pun intended - see ?filter).
>
> HTH,

Thanks, Charles, for all the helpful pointers!   For the moment, I'll
leave parallelization aside, and will explore using 'crossprod' and
'filter'.   Although, from your suggestion that 8 GB of memory should
be sufficient if I went the parallel, I also wonder whether I'm
suffering not just from inefficient use of computing resources, but
that there's a memory leak as well:   The original 'apply' code would,
in much less than a minute, take over the full 18 GB of memory
available on my workstation, and then leave it functioning at a crawl
for at least a half hour or so.   I'll ask about this by reposting
this message again with a different subject, so no need to address it
in this thread.

Thanks again,
David