# ElasticSearch all Search Threads are waiting on take() method **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2018, 3:13pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918 "2018-11-08T15:13:51Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Sekhar1](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Post date:** [November 8, 2018, 3:13pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/1 "2018-11-08T15:13:51Z")
Hi Team, ElasticSearch : 6.0.0 Operating System : CentOS 7 **JVM arguments [-Xms4g, -Xmx4g, -XX:MaxMetaspaceSize=512m, -XX:+UseParNewGC, -XX:NewSize=128m, -XX:MaxNewSize=128m, -XX:SurvivorRatio=1024, -XX:MaxGCPauseMillis=1000, -XX:+UseCompressedOops, -XX:ParallelGCThreads=8, -XX:ConcGCThreads=8, -XX:+DisableExplicitGC, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=50, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/home/somasekhar.k/softwares/ES6/logs/ESHeapDump.hprof, -XX:+PrintGCDetails, -XX:+PrintGCTimeStamps, -Xloggc:/home/somasekhar.k/softwares/ES6/logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=128M, -Des.path.home=/home/somasekhar.k/softwares/ES6, -Des.path.conf=/home/somasekhar.k/softwares/ES6/config** Index Mapping Details : **{"2\_1024": {"aliases": {},"mappings": {"2\_1024": {"properties": {1: {"type": "text"}, 2: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 4: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 5: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 6: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 7: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 8: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 9: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 10: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 11: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 12: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 13: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 14: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 15: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, 16: {"type": "text","fields": {"exactMatch": {"type": "keyword"},"phoneticMatch": {"type": "text","analyzer": "phonetic\_analyzer"},"subStringMatch": {"type": "text","analyzer": "substring\_analyzer"}}}, "@timestamp": {"type": "date"},"@version": {"type": "text","fields": {"keyword": {"type": "keyword","ignore\_above": 256}}}, "query": {"properties": {"term": {"properties": {7: {"properties": {"exactMatch": {"type": "text","fields": {"keyword": {"type": "keyword","ignore\_above": 256}}}}}}}}},"size": {"type": "long"},"type": {"type": "text","fields": {"keyword": {"type": "keyword","ignore\_above": 256}}}}}},"settings": {"index": {"number\_of\_shards": "1","provided\_name": "2\_1024","creation\_date": "1541681596965","analysis": {"filter": {"phonetic\_filter": {"type": "phonetic","encoder": "double\_metaphone"}},"analyzer": {"phonetic\_analyzer": {"filter": ["phonetic\_filter","lowercase"],"tokenizer": "standard"},"substring\_analyzer": {"filter": "lowercase","tokenizer": "substring\_tokenizer"}},"tokenizer": {"substring\_tokenizer": {"token\_chars": ["letter","digit"],"min\_gram": "3","type": "ngram","max\_gram": "3"}}},"number\_of\_replicas": "0","uuid": "aFpNgjSXQA2eaepVYbjylg","version": {"created": "6000099"}}}}}** All search threads(13) are waiting same place.I taken jstack multiple times but every time showing same place.Please refer below Jstack snapshot ![Screenshot%20from%202018-11-08%2020-09-52](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8bace7ef10edc1b91b6099af07e3d16cbcd7ab93.png) **No of Nodes : 1** **No of Index in ES : 2** **Each Index Size : 1GB** **No of Shards for Index : 1** **No of Document in each Index : 1Million** **No of search requests 1 Million using Low level rest client and doing exact match** **Query : {"size":0,"\_source":false,"query":{"term":{"7.exactMatch":"Subex\_\_www200"}}}** I can't add more nodes to cluster.I increased no of shards to 2,3 and 5 but no improve on speed wise. **I checked Disk I/O statistics also but IOWaiting is o.13% only** **Network Speed : 100M/sec** **I tried Shard Request Cache,Indices queries Cache,Decreasing Semants from 15 to 1 and Pre-loading data into the file system cache but no improvement on search speed.** I need to know why search threads waiting same place and how to overcome this problem **_I am try achieve latency and throughput using single client and single node._** ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 8, 2018, 6:14pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/2 "2018-11-08T18:14:27Z")
Please don't post images of text, it makes them hard to read, impossible to search, and makes it hard for those of us using screenreaders. A thread that's blocked on `sun.misc.Unsafe.park` like this is idle, waiting for some work to do. ---
**Author:** ![Sekhar1](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Post date:** [November 8, 2018, 6:27pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/3 "2018-11-08T18:27:38Z")
Thanks for replay. Try to upload jstack log file but its not accept .log format. Currently I am process 5555 request per sec for basic query but my expectations 10000 request per sec. Can you please explain which work try to do?.Can you please tell how to overcome this problem.If you need any logs I will share ---
**Author:** ![Sekhar1](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Post date:** [November 8, 2018, 6:36pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/4 "2018-11-08T18:36:45Z")
One more thing: index.search.slowlog file showing search query time in micro sec but search fetch time in milliseconds. Is their any way reduce search fetch time? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 8, 2018, 6:44pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/5 "2018-11-08T18:44:54Z")
> [@Sekhar1](#): > > Currently I am process 5555 request per sec for basic query but my expectations 10000 request per sec. > Can you please explain which work try to do?.Can you please tell how to overcome this problem.If you need any logs I will share Given that Elasticsearch itself seems to be mostly idle, my first guess would be that the client is not sending requests fast enough. I'd recommend using [Rally](https://github.com/elastic/rally) to benchmark your cluster to rule out any client-side problems first. ---
**Author:** ![Sekhar1](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Post date:** [November 9, 2018, 12:13am UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/6 "2018-11-09T00:13:29Z")
Thanks for Reply, I am using Low level rest client for sending request in loop and using performRequest() its **sync** method.For each every request waiting for response.Client sending request but ES is taking to time process those request.If ES give response fast then client can able to send request also fast. Note : I need to reduce search processing time in ES.If I able do than my Client very f I try to use Rally for benchmarking but i am getting below error while starting time. I my system already JAVA8 available but its telling JAVA11.I am unable to understand root cause of the problem. Rally|690x387 Logs : **2018-11-09 05:27:28,98 ActorAddr-(T|:20540)/PID:25348 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:28682)] and will forward it now.** **2018-11-09 05:27:28,94 ActorAddr-(T|:9331)/PID:25620 esrally.actor ERROR Cannot process message [\]** **Traceback (most recent call last):** \*\* File "/home/somasekhar.k/softwares/rally/esrally/mechanic/mechanic.py", line 545, in receiveMsg\_StartNodes\*\* \*\* msg.distribution, msg.external, msg.docker)\*\* \*\* File "/home/somasekhar.k/softwares/rally/esrally/mechanic/mechanic.py", line 636, in create\*\* \*\* s = supplier.create(cfg, sources, distribution, build, challenge\_root\_path, car, plugins)\*\* \*\* File "/home/somasekhar.k/softwares/rally/esrally/mechanic/supplier.py", line 27, in create\*\* \*\* java\_home = \_java\_home(car)\*\* \*\* File "/home/somasekhar.k/softwares/rally/esrally/mechanic/supplier.py", line 81, in \_java\_home\*\* \*\* \_, path = jvm.resolve\_path(int(build\_jdk))\*\* \*\* File "/home/somasekhar.k/softwares/rally/esrally/utils/jvm.py", line 94, in resolve\_path\*\* \*\* return majors, \_resolve\_single\_path(majors, sysprop\_reader=sysprop\_reader)\*\* \*\* File "/home/somasekhar.k/softwares/rally/esrally/utils/jvm.py", line 139, in \_resolve\_single\_path\*\* \*\* format(specific\_env\_var, generic\_env\_var, major))\*\* **esrally.exceptions.SystemSetupError: ('Neither JAVA11\_HOME nor JAVA\_HOME point to a JDK 11 installation.', None)** **2018-11-09 05:27:28,100 -not-actor-/PID:25177 esrally.racecontrol ERROR A benchmark failure has occurred** **2018-11-09 05:27:28,100 -not-actor-/PID:25177 esrally.racecontrol INFO Telling benchmark actor to exit.** **2018-11-09 05:27:28,101 -not-actor-/PID:25177 esrally.rally INFO Attempting to shutdown internal actor system.** **2018-11-09 05:27:28,101 ActorAddr-(T|:20540)/PID:25348 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).** **2018-11-09 05:27:28,102 ActorAddr-(T|:28682)/PID:25485 esrally.actor INFO MechanicActor#receiveMessage unrecognized(msg = [\] sender = [ActorAddr-(T|:20540)])** **2018-11-09 05:27:28,102 ActorAddr-(T|:20540)/PID:25348 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).** **2018-11-09 05:27:28,103 ActorAddr-(T|:14120)/PID:25619 esrally.actor INFO mechanic.Dispatcher#receiveMessage unrecognized(msg = [\] sender = [ActorAddr-(T|:28682)])** **2018-11-09 05:27:28,105 ActorAddr-(T|:14120)/PID:25619 esrally.actor INFO mechanic.Dispatcher#receiveMessage unrecognized(msg = [\] sender = [ActorAddr-(T|:9331)])** **2018-11-09 05:27:28,106 ActorAddr-(T|:28682)/PID:25485 esrally.actor ERROR Child actor exited with [ChildActorExited:ActorAddr-(T|:14120)] while in status [starting].** **2018-11-09 05:27:28,109 ActorAddr-(T|:20540)/PID:25348 esrally.actor INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:28682)] (ignoring).** **2018-11-09 05:27:28,112 -not-actor-/PID:25347 root INFO ActorSystem Logging Shutdown** **2018-11-09 05:27:28,115 -not-actor-/PID:25346 root INFO ---- Actor System shutdown** **2018-11-09 05:27:28,116 -not-actor-/PID:25177 esrally.rally INFO Actor system is still running. Waiting...** **2018-11-09 05:27:29,117 -not-actor-/PID:25177 esrally.rally INFO Shutdown completed.** **2018-11-09 05:27:29,117 -not-actor-/PID:25177 esrally.rally ERROR Cannot run subcommand [race].** **Traceback (most recent call last):** **File "/home/somasekhar.k/softwares/rally/esrally/rally.py", line 454, in dispatch\_sub\_command** **race(cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/rally.py", line 383, in race** **with\_actor\_system(lambda c: racecontrol.run(c), cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/rally.py", line 404, in with\_actor\_system** **runnable(cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/rally.py", line 383, in** **with\_actor\_system(lambda c: racecontrol.run(c), cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/racecontrol.py", line 390, in run** **raise e** **File "/home/somasekhar.k/softwares/rally/esrally/racecontrol.py", line 387, in run** **pipeline(cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/racecontrol.py", line 61, in call** **self.target(cfg)** **File "/home/somasekhar.k/softwares/rally/esrally/racecontrol.py", line 314, in from\_sources\_complete** **return race(cfg, sources=True, build=True)** **File "/home/somasekhar.k/softwares/rally/esrally/racecontrol.py", line 286, in race** **raise exceptions.RallyError(result.message, result.cause)** **esrally.exceptions.RallyError: (Neither JAVA11\_HOME nor JAVA\_HOME point to a JDK 11 installation., 'Traceback (most recent call last):\n File "/home/somasekhar.k/softwares/rally/esrally/mechanic/mechanic.py", line 545, in receiveMsg\_StartNodes\n msg.distribution, msg.external, msg.docker)\n File "/home/somasekhar.k/softwares/rally/esrally/mechanic/mechanic.py", line 636, in create\n s = supplier.create(cfg, sources, distribution, build, challenge\_root\_path, car, plugins)\n File "/home/somasekhar.k/softwares/rally/esrally/mechanic/supplier.py", line 27, in create\n java\_home = \_java\_home(car)\n File "/home/somasekhar.k/softwares/rally/esrally/mechanic/supplier.py", line 81, in \_java\_home\n \_, path = jvm.resolve\_path(int(build\_jdk))\n File "/home/somasekhar.k/softwares/rally/esrally/utils/jvm.py", line 94, in resolve\_path\n return majors, \_resolve\_single\_path(majors, sysprop\_reader=sysprop\_reader)\n File "/home/somasekhar.k/softwares/rally/esrally/utils/jvm.py", line 139, in \_resolve\_single\_path\n format(specific\_env\_var, generic\_env\_var, major))\nesrally.exceptions.SystemSetupError: ('Neither JAVA11\_HOME nor JAVA\_HOME point to a JDK 11 installation.', None)\n')** Note : I don't think so Low level rest client is an problem. Please tell me how to upload logs files to [discuss.elastic.co](http://discuss.elastic.co) website. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 9, 2018, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/7 "2018-11-09T09:36:50Z")
> [@Sekhar1](#): > > I am using Low level rest client for sending request in loop and using performRequest() its **sync** method.For each every request waiting for response. This means you're only sending one request at once, which seems unrealistic, and explains why Elasticsearch is hardly doing any work. I think you will get better performance sending requests in parallel. > [@Sekhar1](#): > > I try to use Rally for benchmarking but i am getting below error while starting time. I my system already JAVA8 available but its telling JAVA11.I am unable to understand root cause of the problem. I would ask this question in a separate thread with the `Rally` tag, because I am not a Rally expert. Java 11 is required for Elasticsearch development and perhaps this includes running Rally. The simplest thing to do would be to install Java 11. Please could you format logs using the `` button and check they look ok with the preview before sending. ---
**Author:** ![Sekhar1](https://avatars.discourse-cdn.com/v4/letter/s/e19adc/32.png) [@Sekhar1](https://discuss.elastic.co/u/Sekhar1)\ **Post date:** [November 9, 2018, 10:27am UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/8 "2018-11-09T10:27:32Z")
Thanks for you reply, I sent 3 parallel request from 3 low level rest client but I get not better performance.Still My all threads waiting same place.Please see log ``` `12.0% (119.6ms out of 1s) cpu usage by thread 'elasticsearch[ESNode][search][T#2]' 10/10 snapshots sharing following 10 elements sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:737) java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:647) java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1269) org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:161) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748)` ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [November 9, 2018, 1:44pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/9 "2018-11-09T13:44:38Z")
I do not understand the question. These threads are idle. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 7, 2018, 1:44pm UTC](https://discuss.elastic.co/t/elasticsearch-all-search-threads-are-waiting-on-take-method/155918/10 "2018-12-07T13:44:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.