The active thread always in one dataNode,other dataNodes have very few active thread or no active thread

I have 3 master nodes,6 data nodes and every dataNode has almost same quantity and size data,I export data to dataNode evenly。
when I send requests to do pressure test(my requests is also random) ,I user the command 'GET /_cat/thread_pool/search?v&h=port,ip,id,name,active,queue,rejected,completed' in kibana dashboard ,I find most 'active' in one dataNode ,even 'queue' has number,but other dataNodes have very few active thread or no active thread.Anybody had such situation?

Welcome to our community! :smiley:

How are you doing the tests exactly?

just use jmeter to send request to a springboot program,the paramerter is random .

Do you send the requests to a single node, or to all node evenly?

I send the requests to 3 master node, I also send the requests to some data node,but the result is same

Ok, it'd be useful if you could share the output from the _cat API that you are seeing. Please don't post an image of it though.

1.the command is 'GET /_cat/thread_pool/search?v&h=port,ip,id,name,active,queue,rejected,completed'

2.I had changed my nodes ,but the result is just like this

9300 10.17.107.38 IMEntercTkK0XD8Giked7g search 8 0 0 38763
9300 10.17.107.56 2byzeSwjQSat9pBlAbe0DA search 7 0 0 39184
9300 10.17.107.36 ynuFvHGSTVmlb58FpYO9ig search 32 1 0 72920
9300 10.17.107.37 MsEnlGbxQIK5gkJeQPcr4Q search 8 0 0 117160
9300 10.17.107.39 e_mR6vgcTLq3oI00KQtkcQ search 8 0 0 69293

as you can see , 10.17.107.36 has many 'active' ,even has 'thread',and it has high cpu

What is the output from the _cluster/stats?pretty&human API?

What does your shard allocation look like?

1.Every dataNode has one shard , all the data volume is about 90GB.
2.now my cluster has 5 node ,the master is elected by the ES's election mechanism ,but the result is also same as before , the '_cluster/stats?pretty&human' result:

{
  "_nodes" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "cluster_name" : "master-cluster",
  "cluster_uuid" : "MH40olNXQkWfCRUi_spkJA",
  "timestamp" : 1646271656813,
  "status" : "green",
  "indices" : {
    "count" : 8,
    "shards" : {
      "total" : 25,
      "primaries" : 20,
      "replication" : 0.25,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 5,
          "avg" : 3.125
        },
        "primaries" : {
          "min" : 1,
          "max" : 5,
          "avg" : 2.5
        },
        "replication" : {
          "min" : 0.0,
          "max" : 1.0,
          "avg" : 0.625
        }
      }
    },
    "docs" : {
      "count" : 857997276,
      "deleted" : 4476
    },
    "store" : {
      "size" : "48.7gb",
      "size_in_bytes" : 52326566959
    },
    "fielddata" : {
      "memory_size" : "3.3kb",
      "memory_size_in_bytes" : 3384,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "62.6mb",
      "memory_size_in_bytes" : 65677952,
      "total_count" : 628200706,
      "hit_count" : 47687,
      "miss_count" : 628153019,
      "cache_size" : 24,
      "cache_count" : 5393,
      "evictions" : 5369
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 267,
      "memory" : "171.6mb",
      "memory_in_bytes" : 180011401,
      "terms_memory" : "148.1mb",
      "terms_memory_in_bytes" : 155379717,
      "stored_fields_memory" : "15.5mb",
      "stored_fields_memory_in_bytes" : 16355808,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "1.5kb",
      "norms_memory_in_bytes" : 1536,
      "points_memory" : "7.5mb",
      "points_memory_in_bytes" : 7953472,
      "doc_values_memory" : "313.3kb",
      "doc_values_memory_in_bytes" : 320868,
      "index_writer_memory" : "3.8mb",
      "index_writer_memory_in_bytes" : 4071472,
      "version_map_memory" : "0b",
      "version_map_memory_in_bytes" : 0,
      "fixed_bit_set" : "102.3mb",
      "fixed_bit_set_memory_in_bytes" : 107275864,
      "max_unsafe_auto_id_timestamp" : 1646265608049,
      "file_sizes" : { }
    }
  },
  "nodes" : {
    "count" : {
      "total" : 5,
      "coordinating_only" : 0,
      "data" : 5,
      "ingest" : 5,
      "master" : 5,
      "ml" : 5,
      "voting_only" : 0
    },
    "versions" : [
      "7.5.2"
    ],
    "os" : {
      "available_processors" : 160,
      "allocated_processors" : 160,
      "names" : [
        {
          "name" : "Linux",
          "count" : 5
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "CentOS Linux 7 (Core)",
          "count" : 5
        }
      ],
      "mem" : {
        "total" : "313.7gb",
        "total_in_bytes" : 336843325440,
        "free" : "73.7gb",
        "free_in_bytes" : 79195766784,
        "used" : "239.9gb",
        "used_in_bytes" : 257647558656,
        "free_percent" : 24,
        "used_percent" : 76
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 0
      },
      "open_file_descriptors" : {
        "min" : 711,
        "max" : 735,
        "avg" : 717
      }
    },
    "jvm" : {
      "max_uptime" : "19.6h",
      "max_uptime_in_millis" : 70712917,
      "versions" : [
        {
          "version" : "13.0.1",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "13.0.1+9",
          "vm_vendor" : "AdoptOpenJDK",
          "bundled_jdk" : true,
          "using_bundled_jdk" : true,
          "count" : 5
        }
      ],
      "mem" : {
        "heap_used" : "61.6gb",
        "heap_used_in_bytes" : 66167834992,
        "heap_max" : "150gb",
        "heap_max_in_bytes" : 161061273600
      },
      "threads" : 1206
    },
    "fs" : {
      "total" : "2.4tb",
      "total_in_bytes" : 2641547755520,
      "free" : "2.3tb",
      "free_in_bytes" : 2574707974144,
      "available" : "2.2tb",
      "available_in_bytes" : 2440406626304
    },
    "plugins" : [ ],
    "network_types" : {
      "transport_types" : {
        "security4" : 5
      },
      "http_types" : {
        "security4" : 5
      }
    },
    "discovery_types" : {
      "zen" : 5
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "tar",
        "count" : 5
      }
    ]
  }
}

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

{"_nodes":{"total":5,"successful":5,"failed":0},"cluster_name":"master-cluster","cluster_uuid":"MH40olNXQkWfCRUi_spkJA","timestamp":1646271656813,"status":"green","indices":{"count":8,"shards":{"total":25,"primaries":20,"replication":0.25,"index":{"shards":{"min":2,"max":5,"avg":3.125},"primaries":{"min":1,"max":5,"avg":2.5},"replication":{"min":0,"max":1,"avg":0.625}}},"docs":{"count":857997276,"deleted":4476},"store":{"size":"48.7gb","size_in_bytes":52326566959},"fielddata":{"memory_size":"3.3kb","memory_size_in_bytes":3384,"evictions":0},"query_cache":{"memory_size":"62.6mb","memory_size_in_bytes":65677952,"total_count":628200706,"hit_count":47687,"miss_count":628153019,"cache_size":24,"cache_count":5393,"evictions":5369},"completion":{"size":"0b","size_in_bytes":0},"segments":{"count":267,"memory":"171.6mb","memory_in_bytes":180011401,"terms_memory":"148.1mb","terms_memory_in_bytes":155379717,"stored_fields_memory":"15.5mb","stored_fields_memory_in_bytes":16355808,"term_vectors_memory":"0b","term_vectors_memory_in_bytes":0,"norms_memory":"1.5kb","norms_memory_in_bytes":1536,"points_memory":"7.5mb","points_memory_in_bytes":7953472,"doc_values_memory":"313.3kb","doc_values_memory_in_bytes":320868,"index_writer_memory":"3.8mb","index_writer_memory_in_bytes":4071472,"version_map_memory":"0b","version_map_memory_in_bytes":0,"fixed_bit_set":"102.3mb","fixed_bit_set_memory_in_bytes":107275864,"max_unsafe_auto_id_timestamp":1646265608049,"file_sizes":{}}},"nodes":{"count":{"total":5,"coordinating_only":0,"data":5,"ingest":5,"master":5,"ml":5,"voting_only":0},"versions":["7.5.2"],"os":{"available_processors":160,"allocated_processors":160,"names":[{"name":"Linux","count":5}],"pretty_names":[{"pretty_name":"CentOS Linux 7 (Core)","count":5}],"mem":{"total":"313.7gb","total_in_bytes":336843325440,"free":"73.7gb","free_in_bytes":79195766784,"used":"239.9gb","used_in_bytes":257647558656,"free_percent":24,"used_percent":76}},"process":{"cpu":{"percent":0},"open_file_descriptors":{"min":711,"max":735,"avg":717}},"jvm":{"max_uptime":"19.6h","max_uptime_in_millis":70712917,"versions":[{"version":"13.0.1","vm_name":"OpenJDK 64-Bit Server VM","vm_version":"13.0.1+9","vm_vendor":"AdoptOpenJDK","bundled_jdk":true,"using_bundled_jdk":true,"count":5}],"mem":{"heap_used":"61.6gb","heap_used_in_bytes":66167834992,"heap_max":"150gb","heap_max_in_bytes":161061273600},"threads":1206},"fs":{"total":"2.4tb","total_in_bytes":2641547755520,"free":"2.3tb","free_in_bytes":2574707974144,"available":"2.2tb","available_in_bytes":2440406626304},"plugins":[],"network_types":{"transport_types":{"security4":5},"http_types":{"security4":5}},"discovery_types":{"zen":5},"packaging_types":[{"flavor":"default","type":"tar","count":5}]}}

just format it by a JSON tool :grinning:

It's heaps easier if it's multiline, hence the ?pretty&human part of the call.
You can also just edit that above post, highlight the json and then use the </> (aka code formatting) button. Or markdown style backticks.

OK ,I have done it :grinning: Could you read it.
My English is bad , hope you can understand what I mean :joy:
thanks !

1 Like

This is EOL and you should upgrade as a matter of urgency. Latest is 8.0.1, or 7.17.1.
Also make sure you upgrade your JVM :slight_smile:

Everything looks ok there, so can you show _cat/allocation?v?

shards disk.indices disk.used disk.avail disk.total disk.percent host         ip           node
     5        9.7gb    37.2gb    454.7gb      492gb            7 10.17.107.38 10.17.107.38 dataNode-5
     5        9.7gb    37.1gb    454.8gb      492gb            7 10.17.107.56 10.17.107.56 dataNode-2
     5        9.7gb    37.6gb    454.3gb      492gb            7 10.17.107.36 10.17.107.36 dataNode-3
     5        9.7gb    36.3gb    455.6gb      492gb            7 10.17.107.39 10.17.107.39 dataNode-6
     5        9.7gb    38.8gb    453.2gb      492gb            7 10.17.107.37 10.17.107.37 dataNode-4

By the way ,does 7.5.2 has such question ?
And what does the 'EOL' mean ?

7.5 is no longer supported. Please see the link I provided for more information.

OK,even though in such version, is there any way to deal the question :rofl:

Yep, it's definitely something important to address however.

What does the hot threads on that node show?

1.the command is 'GET _nodes/dataNode-3/hot_threads'
2.the result:

::: {dataNode-3}{ynuFvHGSTVmlb58FpYO9ig}{RyLPV9FwShCMpp7vPIlqHQ}{10.17.107.36}{10.17.107.36:9300}{dilm}{ml.machine_memory=67368665088, ml.max_open_jobs=20, xpack.installed=true}
   Hot threads at 2022-03-03T03:13:33.289Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   98.2% (490.7ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#21]'
     unique snapshot
       app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.getFrame(SegmentTermsEnum.java:213)
       app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.pushFrame(SegmentTermsEnum.java:239)
       app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:540)
       app//org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:124)
       app//org.apache.lucene.index.TermStates.get(TermStates.java:187)
       app//org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:134)
       app//org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:109)
       app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:147)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:729)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132)
       app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181)
       app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source)
       app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146)
       app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13.0.1/java.lang.Thread.run(Thread.java:830)
   
   93.5% (467.3ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#10]'
     unique snapshot
       app//org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame.loadBlock(SegmentTermsEnumFrame.java:204)
       app//org.apache.lucene.codecs.blocktree.SegmentTermsEnum.seekExact(SegmentTermsEnum.java:509)
       app//org.apache.lucene.index.TermStates.loadTermsEnum(TermStates.java:124)
       app//org.apache.lucene.index.TermStates.get(TermStates.java:187)
       app//org.apache.lucene.search.TermQuery$TermWeight.getTermsEnum(TermQuery.java:134)
       app//org.apache.lucene.search.TermQuery$TermWeight.scorer(TermQuery.java:109)
       app//org.apache.lucene.search.Weight.scorerSupplier(Weight.java:147)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:744)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132)
       app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181)
       app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source)
       app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146)
       app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13.0.1/java.lang.Thread.run(Thread.java:830)
   
   92.8% (463.8ms out of 500ms) cpu usage by thread 'elasticsearch[dataNode-3][search][T#31]'
     2/10 snapshots sharing following 31 elements
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.BooleanWeight.scorerSupplier(BooleanWeight.java:379)
       app//org.apache.lucene.search.LRUQueryCache$CachingWrapperWeight.scorerSupplier(LRUQueryCache.java:712)
       app//org.elasticsearch.indices.IndicesQueryCache$CachingWeightWrapper.scorerSupplier(IndicesQueryCache.java:157)
       app//org.apache.lucene.search.ConstantScoreQuery$1.scorerSupplier(ConstantScoreQuery.java:130)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorerSupplier(ToParentBlockJoinQuery.java:143)
       app//org.apache.lucene.search.join.ToParentBlockJoinQuery$BlockJoinWeight.scorer(ToParentBlockJoinQuery.java:132)
       app//org.apache.lucene.search.Weight.bulkScorer(Weight.java:181)
       app//org.elasticsearch.search.internal.ContextIndexSearcher$1.bulkScorer(ContextIndexSearcher.java:162)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchInternal(ContextIndexSearcher.java:189)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:173)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:442)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:270)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:113)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:335)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:355)
       app//org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340)
       app//org.elasticsearch.search.SearchService$$Lambda$4514/0x000000080188fc40.apply(Unknown Source)
       app//org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:146)
       app//org.elasticsearch.action.ActionListener$$Lambda$3811/0x000000080168c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63)
       app//org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
       app//org.elasticsearch.action.ActionRunnable$$Lambda$4528/0x000000080188c440.accept(Unknown Source)
       app//org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13.0.1/java.lang.Thread.run(Thread.java:830)
     8/10 snapshots sharing following 10 elements
       java.base@13.0.1/jdk.internal.misc.Unsafe.park(Native Method)
       java.base@13.0.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
       java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.awaitMatch(LinkedTransferQueue.java:743)
       java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.java:684)
       java.base@13.0.1/java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.java:1366)
       app//org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(SizeBlockingQueue.java:165)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
       java.base@13.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
       java.base@13.0.1/java.lang.Thread.run(Thread.java:830)