r - Foreach %dopar% operator incorrectly load balancing -
am using r version 3.1.2 rstudio-server 0.98.113 on debian build 3.2.0-4-amd64 #1 smp debian 3.2.68-1+deb7u1 x86_64 gnu/linux.
i use %dopar%
operator in foreach
package run code in parallel. however, other use on box seemingly installed few items , %dopar%
use far more number of cores specifying , seems load balance between of them. issue doesn't seem perform non-trivial tasks @ anymore.
this example of testing code i've been using testing %dopar%
loop.
library(iterators) library(foreach) library(doparallel) library(parallel) ncores <- 4 cl <- makecluster(ncores) registerdoparallel(cl) trials = 100000 x <- iris[which(iris[,5] != "setosa"),c(1,5)] t2 <- system.time({ r2 <- foreach(icount(trials), .combine=cbind) %dopar% { ind <- sample(100,100,replace= true) results1 <- glm(x[ind,2]~x[ind,1],family=binomial(logit)) coefficients(results1) } })[3] stopcluster(cl)
another interesting behavior can stop r code should running on slave workers had use kill
command handle workers.
i have checked system logs , can't seem find what's been changed it's unclear me how proceed. reminiscent of change in blas library behavior persists after recompiling r not use shared blas lib.
here output lsof -p 23618 | grep 'blas\|lapack'
r 39781 mem reg 8,17 3576576 1055038 /usr/local/lib/r/lib/librlapack.so r 39781 mem reg 8,17 655336 1572936 /usr/lib/libblas/libblas.so.3.0
output r cmd ldd /usr/local/lib/r/bin/exec/r
linux-vdso.so.1 (0x00007ffca59fd000) libr.so => /usr/local/lib/r/lib/libr.so (0x00007f15fdc30000) libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f15fd9f7000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f15fd7da000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f15fd431000) libblas.so.3 => /usr/lib/libblas.so.3 (0x00007f15fd190000) libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3 (0x00007f15fce72000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f15fcb71000) libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0 (0x00007f15fc933000) libreadline.so.6 => /lib/x86_64-linux-gnu/libreadline.so.6 (0x00007f15fc6e9000) liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f15fc4c6000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f15fc2bd000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f15fc0b9000) libicuuc.so.48 => /usr/lib/x86_64-linux-gnu/libicuuc.so.48 (0x00007f15fbd4a000) libicui18n.so.48 => /usr/lib/x86_64-linux-gnu/libicui18n.so.48 (0x00007f15fb97e000) /lib64/ld-linux-x86-64.so.2 (0x00007f15fe1cf000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f15fb768000) libtinfo.so.5 => /lib/x86_64-linux-gnu/libtinfo.so.5 (0x00007f15fb53d000) libicudata.so.48 => /usr/lib/x86_64-linux-gnu/libicudata.so.48 (0x00007f15fa1cd000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f15f9ec2000)
is possible rebuild r without shared library's? open suggestions @ point.
thanks guys!
Comments
Post a Comment