Elastic 8.15.0 slow query if search > 6 days

zenkovac · September 24, 2024, 3:27pm

hi guys, i'm seeing slow queries from kibana if I search for more than 6 days using a wildard search like so: *word*.
If I query for 6 days result is instant at about 3 seconds, but more than 6 days and kibana just hangs loading for about 100 seconds.

I'm migrating from elk 7.16.3 to 8.15 so I have 2 running clusters in parallel and can compare the two in performance, and noticed this behaviour, this doesn't happen in 7.16.3.

Both clusters have the same infra:
2 coordinators, 3 master/data, 2 data only.
Elastic nodes have 2TB of SSD disk, 16GB of ram, 10 cpu's Xeon(R) Gold 6152

index is 8gb per day with daily rotation, 1 primary and 1 replica
max retention is 30 days.

any clues?

zenkovac · September 27, 2024, 12:03pm

any ideas where i should be looking to understand this behaviour?

dadoonet · September 27, 2024, 12:19pm

wildcards are super slow in general, as stated in documentation.

Is that happening on the first run or after some queries, it becomes to behave as it was with 7.x?

You could try to diagnose what's happening with the hot threads API while teh query is running and the profile option in the search query.

Otherwise, if wildcards are really needed, you should switch to the wildcard field type if you are not using it yet.

But may be explain your usage with some sample data and requests and we can find a better way to approach this?

zenkovac · September 27, 2024, 1:30pm

hi david, it happens always in version 8, version 7 never has this delays, in both cases behaviour is consistent all the time.
we dont really use wildcard all the time, this index has logs and we sometimes need to do a kibana search with wildcards so me dont miss any docs doing analysis/troubleshooting. What really worried me was having the same infra and data in the index but less performance in version 8 for the same query.

query is very basic is just a wildcard in time range:

GET /my_index/_search
{
  "size": 500,
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "query_string": {
            "query": "*word*"
          }
        },
        {
          "range": {
            "timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2024-09-20T03:00:00.000Z",
              "lte": "2024-09-27T13:04:37.247Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

somehow the above query runs in 3 seconds using console dev, but 100 or more seconds using kibana search web, i noticed kibana uses _async_search that may be something to look at.

this is the hot threads using kibana:

::: {elk-kib01}{xlG41qSTRbStduVhMU9-dw}{rugEHnFDSqC7AZwEK-vnBQ}{elk-kib01}{10.25.128.123}{10.25.128.123:9300}{i}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.807Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-kib02}{cddyXNSWQ2-3P3vMezSiUw}{49jyrlx6Rw6mnwJ156nsbg}{elk-kib02}{10.25.128.124}{10.25.128.124:9300}{i}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node05}{XzGHlrtHRAOlstOXxtg1FA}{AicmTOSEQ_6YFuPpDdxupA}{elk-node05}{10.25.128.127}{10.25.128.127:9300}{d}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node04}{7m0rtzsiTpunJCvUDPet5A}{hg1ReQYMQVuqcDU4tbcxWQ}{elk-node04}{10.25.128.126}{10.25.128.126:9300}{d}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node03}{hP2dBmugSz-g5NtMibJGLQ}{MIKzbBwcSei22WtQg4ASHg}{elk-node03}{10.25.128.122}{10.25.128.122:9300}{dm}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node01}{sLlb1ZzjRXCs0D__4RItDA}{U1VLzr9KSOO9Qu-GBJPvmg}{elk-node01}{10.25.128.120}{10.25.128.120:9300}{dm}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node02}{odwgwiFXRgObisdOzmzA9Q}{hje1oUT2RQysXN_FeWTGZg}{elk-node02}{10.25.128.121}{10.25.128.121:9300}{dm}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elk-node02][search][T#9]'
     2/10 snapshots sharing following 39 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.CompressionAlgorithm$2.read(CompressionAlgorithm.java:37)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:200)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.loadNextFloorBlock(IntersectTermsEnumFrame.java:121)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.popPushNext(IntersectTermsEnum.java:342)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:548)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     3/10 snapshots sharing following 38 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:200)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:151)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:205)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:536)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     2/10 snapshots sharing following 36 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:195)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:536)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     3/10 snapshots sharing following 35 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:519)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
   
    6.2% [cpu=5.1%, other=1.1%] (31ms out of 500ms) cpu usage by thread 'elasticsearch[elk-node02][write][T#9]'
     7/10 snapshots sharing following 3 elements
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)