# The active thread always in one dataNode，other dataNodes have very few active thread or no active thread **URL:** \ **Category:** Elasticsearch\ **Created:** [March 3, 2022, 12:46am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686 "2022-03-03T00:46:25Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 12:46am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/1 "2022-03-03T00:46:25Z")
I have 3 master nodes,6 data nodes and every dataNode has almost same quantity and size data，I export data to dataNode evenly。 when I send requests to do pressure test(my requests is also random) ，I user the command 'GET /\_cat/thread\_pool/search?v&h=port,ip,id,name,active,queue,rejected,completed' in kibana dashboard ,I find most 'active' in one dataNode ,even 'queue' has number,but other dataNodes have very few active thread or no active thread.Anybody had such situation? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 12:47am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/2 "2022-03-03T00:47:56Z")
Welcome to our community! 😃 How are you doing the tests exactly? ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 12:50am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/3 "2022-03-03T00:50:33Z")
just use jmeter to send request to a springboot program,the paramerter is random . ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 12:53am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/4 "2022-03-03T00:53:16Z")
Do you send the requests to a single node, or to all node evenly? ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 12:54am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/5 "2022-03-03T00:54:52Z")
I send the requests to 3 master node, I also send the requests to some data node,but the result is same ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 12:56am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/6 "2022-03-03T00:56:21Z")
Ok, it'd be useful if you could share the output from the `_cat` API that you are seeing. Please don't post an image of it though. ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 1:01am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/7 "2022-03-03T01:01:47Z")
1.the command is 'GET /\_cat/thread\_pool/search?v&h=port,ip,id,name,active,queue,rejected,completed' 2.I had changed my nodes ,but the result is just like this 9300 10.17.107.38 IMEntercTkK0XD8Giked7g search 8 0 0 38763 9300 10.17.107.56 2byzeSwjQSat9pBlAbe0DA search 7 0 0 39184 9300 10.17.107.36 ynuFvHGSTVmlb58FpYO9ig search 32 1 0 72920 9300 10.17.107.37 MsEnlGbxQIK5gkJeQPcr4Q search 8 0 0 117160 9300 10.17.107.39 e\_mR6vgcTLq3oI00KQtkcQ search 8 0 0 69293 as you can see , 10.17.107.36 has many 'active' ,even has 'thread',and it has high cpu ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 1:32am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/8 "2022-03-03T01:32:24Z")
What is the output from the `_cluster/stats?pretty&human` API? What does your shard allocation look like? ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 1:49am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/9 "2022-03-03T01:49:03Z")
1.Every dataNode has one shard , all the data volume is about 90GB. 2.now my cluster has 5 node ,the master is elected by the ES's election mechanism ，but the result is also same as before , the '\_cluster/stats?pretty&human' result: ```auto { "_nodes" : { "total" : 5, "successful" : 5, "failed" : 0 }, "cluster_name" : "master-cluster", "cluster_uuid" : "MH40olNXQkWfCRUi_spkJA", "timestamp" : 1646271656813, "status" : "green", "indices" : { "count" : 8, "shards" : { "total" : 25, "primaries" : 20, "replication" : 0.25, "index" : { "shards" : { "min" : 2, "max" : 5, "avg" : 3.125 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 2.5 }, "replication" : { "min" : 0.0, "max" : 1.0, "avg" : 0.625 } } }, "docs" : { "count" : 857997276, "deleted" : 4476 }, "store" : { "size" : "48.7gb", "size_in_bytes" : 52326566959 }, "fielddata" : { "memory_size" : "3.3kb", "memory_size_in_bytes" : 3384, "evictions" : 0 }, "query_cache" : { "memory_size" : "62.6mb", "memory_size_in_bytes" : 65677952, "total_count" : 628200706, "hit_count" : 47687, "miss_count" : 628153019, "cache_size" : 24, "cache_count" : 5393, "evictions" : 5369 }, "completion" : { "size" : "0b", "size_in_bytes" : 0 }, "segments" : { "count" : 267, "memory" : "171.6mb", "memory_in_bytes" : 180011401, "terms_memory" : "148.1mb", "terms_memory_in_bytes" : 155379717, "stored_fields_memory" : "15.5mb", "stored_fields_memory_in_bytes" : 16355808, "term_vectors_memory" : "0b", "term_vectors_memory_in_bytes" : 0, "norms_memory" : "1.5kb", "norms_memory_in_bytes" : 1536, "points_memory" : "7.5mb", "points_memory_in_bytes" : 7953472, "doc_values_memory" : "313.3kb", "doc_values_memory_in_bytes" : 320868, "index_writer_memory" : "3.8mb", "index_writer_memory_in_bytes" : 4071472, "version_map_memory" : "0b", "version_map_memory_in_bytes" : 0, "fixed_bit_set" : "102.3mb", "fixed_bit_set_memory_in_bytes" : 107275864, "max_unsafe_auto_id_timestamp" : 1646265608049, "file_sizes" : { } } }, "nodes" : { "count" : { "total" : 5, "coordinating_only" : 0, "data" : 5, "ingest" : 5, "master" : 5, "ml" : 5, "voting_only" : 0 }, "versions" : [ "7.5.2" ], "os" : { "available_processors" : 160, "allocated_processors" : 160, "names" : [ { "name" : "Linux", "count" : 5 } ], "pretty_names" : [ { "pretty_name" : "CentOS Linux 7 (Core)", "count" : 5 } ], "mem" : { "total" : "313.7gb", "total_in_bytes" : 336843325440, "free" : "73.7gb", "free_in_bytes" : 79195766784, "used" : "239.9gb", "used_in_bytes" : 257647558656, "free_percent" : 24, "used_percent" : 76 } }, "process" : { "cpu" : { "percent" : 0 }, "open_file_descriptors" : { "min" : 711, "max" : 735, "avg" : 717 } }, "jvm" : { "max_uptime" : "19.6h", "max_uptime_in_millis" : 70712917, "versions" : [ { "version" : "13.0.1", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "13.0.1+9", "vm_vendor" : "AdoptOpenJDK", "bundled_jdk" : true, "using_bundled_jdk" : true, "count" : 5 } ], "mem" : { "heap_used" : "61.6gb", "heap_used_in_bytes" : 66167834992, "heap_max" : "150gb", "heap_max_in_bytes" : 161061273600 }, "threads" : 1206 }, "fs" : { "total" : "2.4tb", "total_in_bytes" : 2641547755520, "free" : "2.3tb", "free_in_bytes" : 2574707974144, "available" : "2.2tb", "available_in_bytes" : 2440406626304 }, "plugins" : [], "network_types" : { "transport_types" : { "security4" : 5 }, "http_types" : { "security4" : 5 } }, "discovery_types" : { "zen" : 5 }, "packaging_types" : [ { "flavor" : "default", "type" : "tar", "count" : 5 } ] } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 1:51am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/10 "2022-03-03T01:51:01Z")
Please format your code/logs/config using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 2:24am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/11 "2022-03-03T02:24:50Z")
```auto {"_nodes":{"total":5,"successful":5,"failed":0},"cluster_name":"master-cluster","cluster_uuid":"MH40olNXQkWfCRUi_spkJA","timestamp":1646271656813,"status":"green","indices":{"count":8,"shards":{"total":25,"primaries":20,"replication":0.25,"index":{"shards":{"min":2,"max":5,"avg":3.125},"primaries":{"min":1,"max":5,"avg":2.5},"replication":{"min":0,"max":1,"avg":0.625}}},"docs":{"count":857997276,"deleted":4476},"store":{"size":"48.7gb","size_in_bytes":52326566959},"fielddata":{"memory_size":"3.3kb","memory_size_in_bytes":3384,"evictions":0},"query_cache":{"memory_size":"62.6mb","memory_size_in_bytes":65677952,"total_count":628200706,"hit_count":47687,"miss_count":628153019,"cache_size":24,"cache_count":5393,"evictions":5369},"completion":{"size":"0b","size_in_bytes":0},"segments":{"count":267,"memory":"171.6mb","memory_in_bytes":180011401,"terms_memory":"148.1mb","terms_memory_in_bytes":155379717,"stored_fields_memory":"15.5mb","stored_fields_memory_in_bytes":16355808,"term_vectors_memory":"0b","term_vectors_memory_in_bytes":0,"norms_memory":"1.5kb","norms_memory_in_bytes":1536,"points_memory":"7.5mb","points_memory_in_bytes":7953472,"doc_values_memory":"313.3kb","doc_values_memory_in_bytes":320868,"index_writer_memory":"3.8mb","index_writer_memory_in_bytes":4071472,"version_map_memory":"0b","version_map_memory_in_bytes":0,"fixed_bit_set":"102.3mb","fixed_bit_set_memory_in_bytes":107275864,"max_unsafe_auto_id_timestamp":1646265608049,"file_sizes":{}}},"nodes":{"count":{"total":5,"coordinating_only":0,"data":5,"ingest":5,"master":5,"ml":5,"voting_only":0},"versions":["7.5.2"],"os":{"available_processors":160,"allocated_processors":160,"names":[{"name":"Linux","count":5}],"pretty_names":[{"pretty_name":"CentOS Linux 7 (Core)","count":5}],"mem":{"total":"313.7gb","total_in_bytes":336843325440,"free":"73.7gb","free_in_bytes":79195766784,"used":"239.9gb","used_in_bytes":257647558656,"free_percent":24,"used_percent":76}},"process":{"cpu":{"percent":0},"open_file_descriptors":{"min":711,"max":735,"avg":717}},"jvm":{"max_uptime":"19.6h","max_uptime_in_millis":70712917,"versions":[{"version":"13.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"13.0.1+9","vm_vendor":"AdoptOpenJDK","bundled_jdk":true,"using_bundled_jdk":true,"count":5}],"mem":{"heap_used":"61.6gb","heap_used_in_bytes":66167834992,"heap_max":"150gb","heap_max_in_bytes":161061273600},"threads":1206},"fs":{"total":"2.4tb","total_in_bytes":2641547755520,"free":"2.3tb","free_in_bytes":2574707974144,"available":"2.2tb","available_in_bytes":2440406626304},"plugins":[],"network_types":{"transport_types":{"security4":5},"http_types":{"security4":5}},"discovery_types":{"zen":5},"packaging_types":[{"flavor":"default","type":"tar","count":5}]}} ``` ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 2:25am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/12 "2022-03-03T02:25:58Z")
just format it by a JSON tool 😀 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 2:31am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/13 "2022-03-03T02:31:40Z")
It's heaps easier if it's multiline, hence the `?pretty&human` part of the call. You can also just edit that above post, highlight the json and then use the `` (aka code formatting) button. Or markdown style backticks. ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 2:39am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/14 "2022-03-03T02:39:11Z")
OK ,I have done it 😀 Could you read it. My English is bad , hope you can understand what I mean 😂 thanks ! ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 2:42am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/15 "2022-03-03T02:42:30Z")
> [@1506299760](#): > > ```auto > "7.5.2" > > ``` This is [EOL](https://www.elastic.co/support/eol) and you should upgrade as a matter of urgency. Latest is 8.0.1, or 7.17.1. Also make sure you upgrade your JVM 🙂 Everything looks ok there, so can you show `_cat/allocation?v`? ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 2:48am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/16 "2022-03-03T02:48:59Z")
```auto shards disk.indices disk.used disk.avail disk.total disk.percent host ip node 5 9.7gb 37.2gb 454.7gb 492gb 7 10.17.107.38 10.17.107.38 dataNode-5 5 9.7gb 37.1gb 454.8gb 492gb 7 10.17.107.56 10.17.107.56 dataNode-2 5 9.7gb 37.6gb 454.3gb 492gb 7 10.17.107.36 10.17.107.36 dataNode-3 5 9.7gb 36.3gb 455.6gb 492gb 7 10.17.107.39 10.17.107.39 dataNode-6 5 9.7gb 38.8gb 453.2gb 492gb 7 10.17.107.37 10.17.107.37 dataNode-4 ``` By the way ,does 7.5.2 has such question ? And what does the 'EOL' mean ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 2:50am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/17 "2022-03-03T02:50:35Z")
> [@1506299760](#): > > And what does the 'EOL' mean ? 7.5 is no longer supported. Please see the link I provided for more information. ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 3:01am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/18 "2022-03-03T03:01:03Z")
OK，even though in such version, is there any way to deal the question 🤣 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 3, 2022, 3:08am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/19 "2022-03-03T03:08:33Z")
Yep, it's definitely something important to address however. What does the hot threads on that node show? ---
**Author:** ![1506299760](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1506299760/32/102537_2.png) [@1506299760](https://discuss.elastic.co/u/1506299760)\ **Post date:** [March 3, 2022, 3:15am UTC](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686/20 "2022-03-03T03:15:33Z")
1.the command is 'GET \_nodes/dataNode-3/hot\_threads' 2.the result: ```auto ::: {dataNode-3}{ynuFvHGSTVmlb58FpYO9ig}{RyLPV9FwShCMpp7vPIlqHQ}{10.17.107.36}{10.17.107.36:9300}{dilm}{ml.machine_memory=67368665088, ml.max_open_jobs=20, xpack.installed=true} Hot threads at 2022-03-03T03:13:33.289Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true: 98.2% (490.7ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#21]' unique snapshot app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:213) app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:239) app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:540) app//org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:124) app//org.apache.lucene.index.TermStates.get(TermStates.java:187) app//org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:134) app//org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:109) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:147) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:729) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source) app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@13.0.1/java.lang.Thread.run(Thread.java:830) 93.5% (467.3ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#10]' unique snapshot app//org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:204) app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:509) app//org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:124) app//org.apache.lucene.index.TermStates.get(TermStates.java:187) app//org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:134) app//org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:109) app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:147) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:744) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source) app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@13.0.1/java.lang.Thread.run(Thread.java:830) 92.8% (463.8ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#31]' 2/10 snapshots sharing following 31 elements app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379) app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712) app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157) app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143) app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132) app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181) app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162) app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189) app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173) app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270) app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113) app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335) app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355) app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source) app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146) app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source) app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58) app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source) app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@13.0.1/java.lang.Thread.run(Thread.java:830) 8/10 snapshots sharing following 10 elements java.base@13.0.1/jdk.internal.misc.Unsafe.park(Native Method) java.base@13.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743) java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684) java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366) app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) java.base@13.0.1/java.lang.Thread.run(Thread.java:830) ``` [Next page](https://discuss.elastic.co/t/the-active-thread-always-in-one-datanode-other-datanodes-have-very-few-active-thread-or-no-active-thread/298686.md?page=2)