R - parallel processing using window core + linux core at the same time

user1486507 Source

I have been using library(multicore) on linux (8 cores) for parallel processing, but it was not fast enough (sigh..) so I'm trying to utilize my window workstation (8 cores, too). (So it will be using 16 cores if I can use both). It seems like foreach, snow, doSMP is used on Window and multicore is frequently used for Linux.

Can anyone share the experience of using both system for parallel processing or point out any good example?

I tried to use SNOW but I couldn't find the relevant packages(Rmpi) on windows version R in order to connect window to linux machine, which makes me think that this could be impossible..

Any advice will be much appreciated!




answered 6 years ago Paul Hiemstra #1

As Joshua already commented, SOCK clusters can contain nodes across operating systems. I've seen this working for a co-worker. See the documentation of makeSOCKcluster for an example of how to do this.

answered 6 years ago psandersen #2

I've used the doRedis packages with foreach across a network of windows, linux and mac machines. It works well and is pretty straightforward to setup.

This should be all you need to get started: http://cran.r-project.org/web/packages/doRedis/vignettes/doRedis.pdf

comments powered by Disqus