multithreading - Elasticsearch transport client spawning a large number of threads -
from apache storm bolt, using elasticsearch's transport client remotely connect es cluster. when take jstack output of storm process, notice there 1000 threads es stack trace like:
elasticsearch[flying tiger][transport_client_worker][t#22]{new i/o worker #269}" daemon prio=10 tid=0x00007f80ac3cb000 nid=0x356b runnable [0x00007f7e96b2a000] java.lang.thread.state: runnable @ sun.nio.ch.epollarraywrapper.epollwait(native method) @ sun.nio.ch.epollarraywrapper.poll(epollarraywrapper.java:269) @ sun.nio.ch.epollselectorimpl.doselect(epollselectorimpl.java:79) @ sun.nio.ch.selectorimpl.lockanddoselect(selectorimpl.java:87) - locked <0x00000000d148d138> (a sun.nio.ch.util$2) - locked <0x00000000d148d128> (a java.util.collections$unmodifiableset) - locked <0x00000000d148c9b8> (a sun.nio.ch.epollselectorimpl) @ sun.nio.ch.selectorimpl.select(selectorimpl.java:98) @ org.elasticsearch.common.netty.channel.socket.nio.selectorutil.select(selectorutil.java:68) @ org.elasticsearch.common.netty.channel.socket.nio.abstractnioselector.select(abstractnioselector.java:415) @ org.elasticsearch.common.netty.channel.socket.nio.abstractnioselector.run(abstractnioselector.java:212) @ org.elasticsearch.common.netty.channel.socket.nio.abstractnioworker.run(abstractnioworker.java:89) @ org.elasticsearch.common.netty.channel.socket.nio.nioworker.run(nioworker.java:178) @ org.elasticsearch.common.netty.util.threadrenamingrunnable.run(threadrenamingrunnable.java:108) @ org.elasticsearch.common.netty.util.internal.deadlockproofworker$1.run(deadlockproofworker.java:42) @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145) @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615) @ java.lang.thread.run(thread.java:745)
i using single instance of es transport client across storm topology, has 18 output streams invoking es client write data es cluster.
why es transport client spawn many threads ? there way can tune ? tried looking es documentation not provide internal details on threading mechanism of transport client nor give option tune number of threads of client.
i had similar experience before. you've mentioned, 1 transport client creates tens of threads including timers , on.
what have check whether there 1 transport client per worker process. in earlier days, when used 32 transport clients, there more 1 thousand threads , after i've correctly modified singleton instance, number of threads decreased less 2 hundreds(including other threads created in topology)
Comments
Post a Comment