Elastic 8.15.0 slow query if search > 6 days

zenkovac · September 24, 2024, 3:27pm

hi guys, i'm seeing slow queries from kibana if I search for more than 6 days using a wildard search like so: *word*.
If I query for 6 days result is instant at about 3 seconds, but more than 6 days and kibana just hangs loading for about 100 seconds.

I'm migrating from elk 7.16.3 to 8.15 so I have 2 running clusters in parallel and can compare the two in performance, and noticed this behaviour, this doesn't happen in 7.16.3.

Both clusters have the same infra:
2 coordinators, 3 master/data, 2 data only.
Elastic nodes have 2TB of SSD disk, 16GB of ram, 10 cpu's Xeon(R) Gold 6152

index is 8gb per day with daily rotation, 1 primary and 1 replica
max retention is 30 days.

any clues?

zenkovac · September 27, 2024, 12:03pm

any ideas where i should be looking to understand this behaviour?

dadoonet · September 27, 2024, 12:19pm

wildcards are super slow in general, as stated in documentation.

Is that happening on the first run or after some queries, it becomes to behave as it was with 7.x?

You could try to diagnose what's happening with the hot threads API while teh query is running and the profile option in the search query.

Otherwise, if wildcards are really needed, you should switch to the wildcard field type if you are not using it yet.

But may be explain your usage with some sample data and requests and we can find a better way to approach this?

zenkovac · September 27, 2024, 1:30pm

hi david, it happens always in version 8, version 7 never has this delays, in both cases behaviour is consistent all the time.
we dont really use wildcard all the time, this index has logs and we sometimes need to do a kibana search with wildcards so me dont miss any docs doing analysis/troubleshooting. What really worried me was having the same infra and data in the index but less performance in version 8 for the same query.

query is very basic is just a wildcard in time range:

GET /my_index/_search
{
  "size": 500,
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "query_string": {
            "query": "*word*"
          }
        },
        {
          "range": {
            "timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2024-09-20T03:00:00.000Z",
              "lte": "2024-09-27T13:04:37.247Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

somehow the above query runs in 3 seconds using console dev, but 100 or more seconds using kibana search web, i noticed kibana uses _async_search that may be something to look at.

this is the hot threads using kibana:

::: {elk-kib01}{xlG41qSTRbStduVhMU9-dw}{rugEHnFDSqC7AZwEK-vnBQ}{elk-kib01}{10.25.128.123}{10.25.128.123:9300}{i}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.807Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-kib02}{cddyXNSWQ2-3P3vMezSiUw}{49jyrlx6Rw6mnwJ156nsbg}{elk-kib02}{10.25.128.124}{10.25.128.124:9300}{i}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node05}{XzGHlrtHRAOlstOXxtg1FA}{AicmTOSEQ_6YFuPpDdxupA}{elk-node05}{10.25.128.127}{10.25.128.127:9300}{d}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node04}{7m0rtzsiTpunJCvUDPet5A}{hg1ReQYMQVuqcDU4tbcxWQ}{elk-node04}{10.25.128.126}{10.25.128.126:9300}{d}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node03}{hP2dBmugSz-g5NtMibJGLQ}{MIKzbBwcSei22WtQg4ASHg}{elk-node03}{10.25.128.122}{10.25.128.122:9300}{dm}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node01}{sLlb1ZzjRXCs0D__4RItDA}{U1VLzr9KSOO9Qu-GBJPvmg}{elk-node01}{10.25.128.120}{10.25.128.120:9300}{dm}{8.15.0}{7000099-8512000}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

::: {elk-node02}{odwgwiFXRgObisdOzmzA9Q}{hje1oUT2RQysXN_FeWTGZg}{elk-node02}{10.25.128.121}{10.25.128.121:9300}{dm}{8.15.0}{7000099-8512000}{xpack.installed=true, ml.config_version=12.0.0, transform.config_version=10.0.0}
   Hot threads at 2024-09-27T13:19:25.808Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   100.0% [cpu=99.8%, other=0.2%] (500ms out of 500ms) cpu usage by thread 'elasticsearch[elk-node02][search][T#9]'
     2/10 snapshots sharing following 39 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.CompressionAlgorithm$2.read(CompressionAlgorithm.java:37)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:200)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.loadNextFloorBlock(IntersectTermsEnumFrame.java:121)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.popPushNext(IntersectTermsEnum.java:342)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:548)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     3/10 snapshots sharing following 38 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:200)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnumFrame.load(IntersectTermsEnumFrame.java:151)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:205)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:536)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     2/10 snapshots sharing following 36 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.pushFrame(IntersectTermsEnum.java:195)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:536)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
     3/10 snapshots sharing following 35 elements
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum._next(IntersectTermsEnum.java:519)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.codecs.lucene90.blocktree.IntersectTermsEnum.next(IntersectTermsEnum.java:377)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.index.FilterLeafReader$FilterTermsEnum.next(FilterLeafReader.java:201)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.internal.ExitableDirectoryReader$ExitableTermsEnum.next(ExitableDirectoryReader.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMatchesIterator.fromTermsEnum(DisjunctionMatchesIterator.java:89)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.lambda$matches$0(AbstractMultiTermQueryConstantScoreWrapper.java:252)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight$$Lambda/0x00007efb58129490.get(Unknown Source)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.MatchesUtils.forField(MatchesUtils.java:115)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.AbstractMultiTermQueryConstantScoreWrapper$RewritingWeight.matches(AbstractMultiTermQueryConstantScoreWrapper.java:249)
       app/org.apache.lucene.core@9.11.1/org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.matches(DisjunctionMaxQuery.java:122)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumsWeightMatcher(FieldOffsetStrategy.java:147)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldOffsetStrategy.createOffsetsEnumFromReader(FieldOffsetStrategy.java:74)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.MemoryIndexOffsetStrategy.getOffsetsEnum(MemoryIndexOffsetStrategy.java:119)
       app/org.apache.lucene.highlighter@9.11.1/org.apache.lucene.search.uhighlight.FieldHighlighter.highlightFieldForDoc(FieldHighlighter.java:83)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomFieldHighlighter.highlightFieldForDoc(CustomFieldHighlighter.java:74)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.lucene.search.uhighlight.CustomUnifiedHighlighter.highlightField(CustomUnifiedHighlighter.java:150)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.DefaultHighlighter.highlight(DefaultHighlighter.java:86)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.subphase.highlight.HighlightPhase$1.process(HighlightPhase.java:69)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase$1.nextDoc(FetchPhase.java:178)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhaseDocsIterator.iterate(FetchPhaseDocsIterator.java:71)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:190)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:80)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService.lambda$executeFetchPhase$10(SearchService.java:908)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.search.SearchService$$Lambda/0x00007efb586d8f90.get(Unknown Source)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:78)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:75)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:100)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)
       app/org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)
   
    6.2% [cpu=5.1%, other=1.1%] (31ms out of 500ms) cpu usage by thread 'elasticsearch[elk-node02][write][T#9]'
     7/10 snapshots sharing following 3 elements
       java.base@22.0.1/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
       java.base@22.0.1/java.lang.Thread.runWith(Thread.java:1583)
       java.base@22.0.1/java.lang.Thread.run(Thread.java:1570)

zenkovac · October 14, 2024, 10:54am

@dadoonet hi david any ideas?

dadoonet · October 14, 2024, 11:29am

Sorry no. Just that "query": "*word*" is one of the worse thing you can run with Elasticsearch. A wild guess is that when you are running that from Kibana, as it's super slow, Kibana is using an async search behind the scene and every x seconds is looking for the status of this search. I did not find any setting which defines when Kibana switches from sync search to async search.

May be someone knows that or you could ask in Kibana channel?

Also, I'd upgrade to 8.15.2 but I don't remember something has changed on this topic between 8.15.0.

Topic		Replies	Views
Problem in Elasticsearch Wildcard filtered Query Elasticsearch	6	2439	September 8, 2017
Wildcard query took 200 seconds with version 8.8 but only a few seconds with 6.3 Elasticsearch	1	196	September 1, 2023
Wildcard queries on date based indexes, searches all indexes? Elasticsearch	1	386	September 4, 2019
The same query is really slow 20% of the time Elasticsearch	11	1598	October 14, 2019
Elasticsearch searches getting slower and slower with time Elasticsearch	3	442	July 8, 2018

Elastic 8.15.0 slow query if search > 6 days

Related topics