Elasticsearch 7.7 , cpu usage is very high

Elastic Stack Elasticsearch
1

Hi,

My server is working on Elasticsearch 7.7,
Recently the CPU usage is reaching 100% when Kibana is running.
Elasticsearch process CPU usage is very high.
image

GET _cat/nodes?v=true&s=cpu:desc

image
image710×43 3.17 KB

GET /_cluster/settings
image

GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb

image
image181×768 1.57 KB

Elasticsesarch.log –

[2023-01-09T11:52:30,371][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [SERVERNAME] [controller/16543] [Main.cc@110] controller (64 bit): Version 7.7.0 (Build a8939d3da43f33) Copyright (c) 2020 Elasticsearch BV
[2023-01-09T11:52:31,217][INFO ][o.e.d.DiscoveryModule    ] [SERVERNAME] using discovery type [zen] and seed hosts providers [settings]
[2023-01-09T11:52:32,140][INFO ][o.e.n.Node               ] [SERVERNAME] initialized
[2023-01-09T11:52:32,140][INFO ][o.e.n.Node               ] [SERVERNAME] starting ...
[2023-01-09T11:52:32,301][INFO ][o.e.t.TransportService   ] [SERVERNAME] publish_address {SERVERIP:9300}, bound_addresses {[::]:9300}
[2023-01-09T11:52:33,045][INFO ][o.e.b.BootstrapChecks    ] [SERVERNAME] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2023-01-09T11:52:33,066][INFO ][o.e.c.c.Coordinator      ] [SERVERNAME] cluster UUID [0lgUB-62Q-aX_8DtMcyItQ]
[2023-01-09T11:52:33,251][INFO ][o.e.c.s.MasterService    ] [SERVERNAME] elected-as-master ([1] nodes joined)[{SERVERNAME}{kAuGDJkjT0y0mmS0egOdbg}{0L2K7X61TRe85DKbO4PwnQ}{SERVERIP}{SERVERIP:9300}{dilmrt}{ml.machine_memory=67386851328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 53, version: 537423, delta: master node changed {previous [], current [{SERVERNAME}{kAuGDJkjT0y0mmS0egOdbg}{0L2K7X61TRe85DKbO4PwnQ}{SERVERIP}{SERVERIP:9300}{dilmrt}{ml.machine_memory=67386851328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}
[2023-01-09T11:52:33,456][INFO ][o.e.c.s.ClusterApplierService] [SERVERNAME] master node changed {previous [], current [{SERVERNAME}{kAuGDJkjT0y0mmS0egOdbg}{0L2K7X61TRe85DKbO4PwnQ}{SERVERIP}{SERVERIP:9300}{dilmrt}{ml.machine_memory=67386851328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]}, term: 53, version: 537423, reason: Publication{term=53, version=537423}
[2023-01-09T11:52:33,522][INFO ][o.e.h.AbstractHttpServerTransport] [SERVERNAME] publish_address {SERVERIP:9200}, bound_addresses {[::]:9200}
[2023-01-09T11:52:33,523][INFO ][o.e.n.Node               ] [SERVERNAME] started
[2023-01-09T11:52:33,847][INFO ][o.e.c.s.ClusterSettings  ] [SERVERNAME] updating [search.max_open_scroll_context] from [500] to [20480]
[2023-01-09T11:52:34,266][INFO ][o.e.l.LicenseService     ] [SERVERNAME] license [6c950fec-0803-40a8-a66a-4cef26bbd139] mode [basic] - valid
[2023-01-09T11:52:34,268][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [SERVERNAME] Active license is now [BASIC]; Security is disabled
[2023-01-09T11:52:34,278][INFO ][o.e.g.GatewayService     ] [SERVERNAME] recovered [864] indices into cluster_state
[2023-01-09T11:52:35,030][WARN ][r.suppressed             ] [SERVERNAME] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:223) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:288) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2023-01-09T11:52:35,030][WARN ][r.suppressed             ] [SERVERNAME] path: /.kibana_task_manager/_count, params: {index=.kibana_task_manager}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:223) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:288) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.7.0.jar:7.7.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.7.0.jar:7.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2023-01-09T11:52:37,553][WARN ][r.suppressed             ] [SERVERNAME] path: /.kibana_task_manager/_count, params: {index=.kibana_task_manager}

Server resources:

12 Processors

64 GB RAM

5T Disk size – 200GB usage

jvm.options file –


-Xms32g

-Xmx32g

################################################################

##Expert settings

################################################################

##

##All settings below this section are considered

##expert settings. Don't tamper with them unless

##you understand what you are doing

##

################################################################

##GC configuration

8-13:-XX:+UseConcMarkSweepGC

8-13:-XX:CMSInitiatingOccupancyFraction=75

8-13:-XX:+UseCMSInitiatingOccupancyOnly

##G1GC Configuration

#NOTE: G1 GC is only supported on JDK version 10 or later

#to use G1GC, uncomment the next two lines and update the version on the

#following three lines to your version of the JDK

#10-13:-XX:-UseConcMarkSweepGC

#10-13:-XX:-UseCMSInitiatingOccupancyOnly

14-:-XX:+UseG1GC

14-:-XX:G1ReservePercent=25

14-:-XX:InitiatingHeapOccupancyPercent=30

##JVM temporary directory

-Djava.io.tmpdir=${ES_TMPDIR}

##heap dumps

#generate a heap dump when an allocation from the Java heap fails

#heap dumps are created in the working directory of the JVM

-XX:+HeapDumpOnOutOfMemoryError

#specify an alternative path for heap dumps; ensure the directory exists and

#has sufficient space

-XX:HeapDumpPath=/var/lib/elasticsearch

#specify an alternative path for JVM fatal error logs

-XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log

##JDK 8 GC logging

8:-XX:+PrintGCDetails

8:-XX:+PrintGCDateStamps

8:-XX:+PrintTenuringDistribution

8:-XX:+PrintGCApplicationStoppedTime

8:-Xloggc:/var/log/elasticsearch/gc.log

8:-XX:+UseGCLogFileRotation

8:-XX:NumberOfGCLogFiles=32

8:-XX:GCLogFileSize=64m

#JDK 9+ GC logging

9-:Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m

-Dlog4j2.formatMsgNoLookups=true

elasticsesarch.yml file –


path.data: /var/lib/elasticsearch

#

#Path to log files:

#

path.logs: /var/log/elasticsearch

#

#----------------------------------- Memory -----------------------------------

#

#Lock the memory on startup:

#

#bootstrap.memory_lock: true

#

#Make sure that the heap size is set to about half the memory available

#on the system and that the owner of the process is allowed to use this

#limit.

#

#Elasticsearch performs poorly when the system is swapping the memory.

#

#---------------------------------- Network -----------------------------------

#

#Set the bind address to a specific IP (IPv4 or IPv6):

#

#network.host: 192.168.0.1

network.host: 0.0.0.0

#

#Set a custom port for HTTP:

#

#http.port: 9200

http.port: 9200

#

#For more information, consult the network module documentation.

#

#--------------------------------- Discovery ----------------------------------

#

#Pass an initial list of hosts to perform discovery when this node is started:

#The default list of hosts is ["127.0.0.1", "[::1]"]

#

#discovery.seed_hosts: ["host1", "host2"]

#

#Bootstrap the cluster using an initial set of master-eligible nodes:

#

cluster.initial_master_nodes: node-1

logstash jvm.options –

-Xms10g

-Xmx10g

I would appreciate your help,
What can i do to improve resource consumption?

Thanks,
Gil

2

Elasticsearch 7.7 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

What is the full output of the cluster stats API?

4

Hi Christian,
Thanks for the response

GET /_cluster/stats -

{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "0lgUB-62Q-aX_8DtMcyItQ",
  "timestamp" : 1673264740675,
  "status" : "red",
  "indices" : {
    "count" : 857,
    "shards" : {
      "total" : 857,
      "primaries" : 857,
      "replication" : 0.0,
      "index" : {
        "shards" : {
          "min" : 1,
          "max" : 1,
          "avg" : 1.0
        },
        "primaries" : {
          "min" : 1,
          "max" : 1,
          "avg" : 1.0
        },
        "replication" : {
          "min" : 0.0,
          "max" : 0.0,
          "avg" : 0.0
        }
      }
    },
    "docs" : {
      "count" : 577318728,
      "deleted" : 1082684
    },
    "store" : {
      "size_in_bytes" : 216685365545
    },
    "fielddata" : {
      "memory_size_in_bytes" : 15176,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size_in_bytes" : 1396889,
      "total_count" : 3093945,
      "hit_count" : 1638424,
      "miss_count" : 1455521,
      "cache_size" : 437,
      "cache_count" : 843,
      "evictions" : 406
    },
    "completion" : {
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 7667,
      "memory_in_bytes" : 121342360,
      "terms_memory_in_bytes" : 99428448,
      "stored_fields_memory_in_bytes" : 8126664,
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory_in_bytes" : 9484544,
      "points_memory_in_bytes" : 0,
      "doc_values_memory_in_bytes" : 4302704,
      "index_writer_memory_in_bytes" : 4549858332,
      "version_map_memory_in_bytes" : 255,
      "fixed_bit_set_memory_in_bytes" : 809016,
      "max_unsafe_auto_id_timestamp" : 1673258084367,
      "file_sizes" : { }
    },
    "mappings" : {
      "field_types" : [
        {
          "name" : "binary",
          "count" : 3,
          "index_count" : 1
        },
        {
          "name" : "boolean",
          "count" : 36,
          "index_count" : 13
        },
        {
          "name" : "date",
          "count" : 928,
          "index_count" : 864
        },
        {
          "name" : "double",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "flattened",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "float",
          "count" : 8,
          "index_count" : 2
        },
        {
          "name" : "geo_point",
          "count" : 4,
          "index_count" : 3
        },
        {
          "name" : "geo_shape",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "half_float",
          "count" : 12,
          "index_count" : 1
        },
        {
          "name" : "integer",
          "count" : 31,
          "index_count" : 4
        },
        {
          "name" : "ip",
          "count" : 2,
          "index_count" : 1
        },
        {
          "name" : "keyword",
          "count" : 17380,
          "index_count" : 864
        },
        {
          "name" : "long",
          "count" : 1478,
          "index_count" : 861
        },
        {
          "name" : "nested",
          "count" : 8,
          "index_count" : 2
        },
        {
          "name" : "object",
          "count" : 3688,
          "index_count" : 864
        },
        {
          "name" : "text",
          "count" : 17152,
          "index_count" : 862
        }
      ]
    },
    "analysis" : {
      "char_filter_types" : [ ],
      "tokenizer_types" : [ ],
      "filter_types" : [ ],
      "analyzer_types" : [ ],
      "built_in_char_filters" : [ ],
      "built_in_tokenizers" : [ ],
      "built_in_filters" : [ ],
      "built_in_analyzers" : [
        {
          "name" : "english",
          "count" : 1,
          "index_count" : 1
        }
      ]
    }
  },
  "nodes" : {
    "count" : {
      "total" : 1,
      "coordinating_only" : 0,
      "data" : 1,
      "ingest" : 1,
      "master" : 1,
      "ml" : 1,
      "remote_cluster_client" : 1,
      "transform" : 1,
      "voting_only" : 0
    },
    "versions" : [
      "7.7.0"
    ],
    "os" : {
      "available_processors" : 12,
      "allocated_processors" : 12,
      "names" : [
        {
          "name" : "Linux",
          "count" : 1
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "Red Hat Enterprise Linux Server 7.7 (Maipo)",
          "count" : 1
        }
      ],
      "mem" : {
        "total_in_bytes" : 67386851328,
        "free_in_bytes" : 604250112,
        "used_in_bytes" : 66782601216,
        "free_percent" : 1,
        "used_percent" : 99
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 76
      },
      "open_file_descriptors" : {
        "min" : 8641,
        "max" : 8641,
        "avg" : 8641
      }
    },
    "jvm" : {
      "max_uptime_in_millis" : 6814851,
      "versions" : [
        {
          "version" : "14",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "14+36",
          "vm_vendor" : "AdoptOpenJDK",
          "bundled_jdk" : true,
          "using_bundled_jdk" : true,
          "count" : 1
        }
      ],
      "mem" : {
        "heap_used_in_bytes" : 15519731712,
        "heap_max_in_bytes" : 48318382080
      },
      "threads" : 141
    },
    "fs" : {
      "total_in_bytes" : 5495416946688,
      "free_in_bytes" : 5268619214848,
      "available_in_bytes" : 5268619214848
    },
    "plugins" : [ ],
    "network_types" : {
      "transport_types" : {
        "security4" : 1
      },
      "http_types" : {
        "security4" : 1
      }
    },
    "discovery_types" : {
      "zen" : 1
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "rpm",
        "count" : 1
      }
    ],
    "ingest" : {
      "number_of_pipelines" : 5,
      "processor_stats" : {
        "date" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        },
        "grok" : {
          "count" : 1406,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 4214
        },
        "gsub" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        },
        "remove" : {
          "count" : 1406,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        },
        "script" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time_in_millis" : 0
        }
      }
    }
  }
}
5

What does the hot threads API show when CPU usage is high?

6 
::: {SERVERNAME}{kAuGDJkjT0y0mmS0egOdbg}{0L2K7X61TRe85DKbO4PwnQ}{SERVERIP}{SERVERIP:9300}{dilmrt}{ml.machine_memory=67386851328, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}
   Hot threads at 2023-01-09T12:08:15.036Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:
   
   98.1% (490.5ms out of 500ms) cpu usage by thread 'elasticsearch[SERVERNAME][search][T#12]'
     7/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:242)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
     3/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:244)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
   
   97.0% (485ms out of 500ms) cpu usage by thread 'elasticsearch[SERVERNAME][search][T#16]'
     7/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:242)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
     2/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:244)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
     unique snapshot
       app//org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:270)
       app//org.apache.lucene.util.PriorityQueue.pop(PriorityQueue.java:175)
       app//org.apache.lucene.search.TopDocsCollector.populateResults(TopDocsCollector.java:64)
       app//org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:159)
       app//org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.java:96)
       app//org.elasticsearch.search.query.TopDocsCollectorContext$SimpleTopDocsCollectorContext$$Lambda$4990/0x00007fac30850c58.get(Unknown Source)
       app//org.elasticsearch.common.util.CachedSupplier.get(CachedSupplier.java:42)
       app//org.elasticsearch.search.query.TopDocsCollectorContext$SimpleTopDocsCollectorContext.newTopDocs(TopDocsCollectorContext.java:305)
       app//org.elasticsearch.search.query.TopDocsCollectorContext$SimpleTopDocsCollectorContext.postProcess(TopDocsCollectorContext.java:319)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:361)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
   
   94.1% (470.4ms out of 500ms) cpu usage by thread 'elasticsearch[SERVERNAME][search][T#6]'
     7/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:242)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
     3/10 snapshots sharing following 24 elements
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:244)
       app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:229)
       app//org.apache.lucene.search.ConstantScoreQuery$ConstantBulkScorer.score(ConstantScoreQuery.java:87)
       app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:56)
       app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:212)
       app//org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:185)
       app//org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:445)
       app//org.elasticsearch.search.query.QueryPhase.searchWithCollector(QueryPhase.java:344)
       app//org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:299)
       app//org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)
       app//org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:361)
       app//org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)
       app//org.elasticsearch.search.SearchService.access$200(SearchService.java:135)
       app//org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)
       app//org.elasticsearch.search.SearchService$2$$Lambda$4903/0x00007fb9772ab900.get(Unknown Source)
       app//org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)
       app//org.elasticsearch.search.SearchService$$Lambda$4904/0x00007fb9772aacb0.run(Unknown Source)
       app//org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
       app//org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)
       app//org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
       java.base@14/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
       java.base@14/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
       java.base@14/java.lang.Thread.run(Thread.java:832)
7

A few questions.

Is this disk SSD or HDD?

This seems too much, there is a recommendation for systems with a lot of memory to keep it between 26 GB and 30 GB, 32 GB may be above the threshold for compressed ordinary object pointers, which can impact performance.

10 GB for logstash may be too much, there is also a recommendation to use it between 4GB and 8 GB, I would reduce it to 4GB and only increase if Logstash start throwing OOM errors.

857 shards on a single-node can also give you some issues, version 7 still follows the old rule of thumb of trying to keep a maximun of 20 shards per GB of HEAP, with 32 GB it would be 640, but you will probably need to decrease the heap to 30 GB, so it would be better to keep a maximun of 600 shards on the node.

Elasticsearch 7.7 is very old and hos not any performance improvement from later versions, I would update it as soon as possible, but first try to reduce the heap for Elasticsearch to 30 GB and the heap for Logstash to 4 GB, this will give more RAM to your system and may help.

Also, you have swap enabled, you should disable your swap.

1 Like
8

Hi leandrojmp,
Thanks for your response

I did the changes, Now there is 30GB for elasticsearch and 4GB for logstash.

I removed shards, Now there is 220 shards on this node,

And I disabled swap

image

But unfortunately CPU usage is still very high,
It happens only when the Kibana process is running.

I have a few more servers and I saw that the Kibana index on this server is larger than others servers.

image
image1204×150 9.44 KB

(On other servers the size of kibana index is between 5MB and 10MB)

Could it have an effect?

Thank you

9

Hi,
Another thing,

I saw that there are a lot of tasks on this server:

Example for tasks from Dev Tools:
(there are more than 140 tasks)

GET _tasks/


{
  "nodes" : {
    "kAuGDJkjT0y0mmS0egOdbg" : {
      "name" : "SERVERNAME",
      "transport_address" : "SERVERIP:9300",
      "host" : "SERVERIP",
      "ip" : "SERVERIP:9300",
      "roles" : [
        "ingest",
        "master",
        "transform",
        "data",
        "remote_cluster_client",
        "ml"
      ],
      "attributes" : {
        "ml.machine_memory" : "67386851328",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "20"
      },
      "tasks" : {
        "kAuGDJkjT0y0mmS0egOdbg:22334036" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334036,
          "type" : "transport",
          "action" : "indices:data/write/bulk[s]",
          "start_time_in_millis" : 1673340030857,
          "running_time_in_nanos" : 50110388,
          "cancellable" : false,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334035",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:2560" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 2560,
          "type" : "transport",
          "action" : "indices:data/read/search",
          "start_time_in_millis" : 1673283901491,
          "running_time_in_nanos" : 56129415609203,
          "cancellable" : true,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:2559",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334037" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334037,
          "type" : "transport",
          "action" : "indices:data/write/bulk[s]",
          "start_time_in_millis" : 1673340030857,
          "running_time_in_nanos" : 50106523,
          "cancellable" : false,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334035",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334038" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334038,
          "type" : "transport",
          "action" : "indices:data/write/bulk[s]",
          "start_time_in_millis" : 1673340030857,
          "running_time_in_nanos" : 50104064,
          "cancellable" : false,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334035",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334039" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334039,
          "type" : "transport",
          "action" : "indices:data/write/bulk[s]",
          "start_time_in_millis" : 1673340030857,
          "running_time_in_nanos" : 50102974,
          "cancellable" : false,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334035",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:1284" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 1284,
          "type" : "transport",
          "action" : "indices:data/read/search",
          "start_time_in_millis" : 1673283878531,
          "running_time_in_nanos" : 56152375761947,
          "cancellable" : true,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:1283",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334032" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334032,
          "type" : "direct",
          "action" : "indices:data/read/search[phase/query]",
          "start_time_in_millis" : 1673340030849,
          "running_time_in_nanos" : 58206883,
          "cancellable" : true,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334031",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334033" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334033,
          "type" : "transport",
          "action" : "indices:data/read/search",
          "start_time_in_millis" : 1673340030849,
          "running_time_in_nanos" : 58109244,
          "cancellable" : true,
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334034" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334034,
          "type" : "direct",
          "action" : "indices:data/read/search[phase/query]",
          "start_time_in_millis" : 1673340030849,
          "running_time_in_nanos" : 58089529,
          "cancellable" : true,
          "parent_task_id" : "kAuGDJkjT0y0mmS0egOdbg:22334033",
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334035" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334035,
          "type" : "transport",
          "action" : "indices:data/write/bulk",
          "start_time_in_millis" : 1673340030857,
          "running_time_in_nanos" : 50283129,
          "cancellable" : false,
          "headers" : { }
        },
        "kAuGDJkjT0y0mmS0egOdbg:22334044" : {
          "node" : "kAuGDJkjT0y0mmS0egOdbg",
          "id" : 22334044,
          "type" : "transport",
          "action" : "indices:data/read/search",
          "start_time_in_millis" : 1673340030868,
          "running_time_in_nanos" : 39489531,
          "cancellable" : true,
          "headers" : { }
        },

When i cancel the tasks the CPU usage is lower.

Maybe this could be the reason?
How can i optimize this?

Thanks

10

Hi,
I have a lot of tasks,
I see that most of the tasks are related to .kibana index:

GET /_cat/tasks?detailed

indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605610 -                              transport 1673435322633 11:08:42 585.8ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":61800,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605614 -                              transport 1673435322635 11:08:42 583.3ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":58600,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605645 -                              transport 1673435322673 11:08:42 545.1ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":4600,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605648 -                              transport 1673435322678 11:08:42 540.2ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":52600,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605650 -                              transport 1673435322678 11:08:42 540.1ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":0,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605682 -                              transport 1673435322686 11:08:42 532ms       SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":34800,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605699 -                              transport 1673435322700 11:08:42 518.4ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":44900,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605708 -                              transport 1673435322716 11:08:42 503.3ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":24100,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605718 -                              transport 1673435322720 11:08:42 498.2ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":33400,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605773 -                              transport 1673435322818 11:08:42 400.6ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":78500,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605774 kAuGDJkjT0y0mmS0egOdbg:3605773 direct    1673435322818 11:08:42 400.5ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605783 -                              transport 1673435322842 11:08:42 376.7ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":7500,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605787 -                              transport 1673435322844 11:08:42 374.9ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":71700,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605788 kAuGDJkjT0y0mmS0egOdbg:3605787 direct    1673435322844 11:08:42 374.9ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605789 -                              transport 1673435322845 11:08:42 373.7ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":18300,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605791 -                              transport 1673435322846 11:08:42 372.5ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":21000,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605792 kAuGDJkjT0y0mmS0egOdbg:3605791 direct    1673435322846 11:08:42 372.5ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605793 -                              transport 1673435322847 11:08:42 371.2ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":9600,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605797 -                              transport 1673435322860 11:08:42 358.8ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":42700,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605801 -                              transport 1673435322872 11:08:42 346.9ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":12300,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605802 kAuGDJkjT0y0mmS0egOdbg:3605801 direct    1673435322872 11:08:42 346.8ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605811 -                              transport 1673435322935 11:08:42 283ms       SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":48600,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605812 kAuGDJkjT0y0mmS0egOdbg:3605811 direct    1673435322935 11:08:42 282.9ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605813 -                              transport 1673435322940 11:08:42 278.1ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":54200,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605814 kAuGDJkjT0y0mmS0egOdbg:3605813 direct    1673435322940 11:08:42 278.1ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605815 -                              transport 1673435322942 11:08:42 276.1ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":66300,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605816 kAuGDJkjT0y0mmS0egOdbg:3605815 direct    1673435322942 11:08:42 276.2ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605817 -                              transport 1673435322943 11:08:42 275ms       SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":30900,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605821 -                              transport 1673435322946 11:08:42 272ms       SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":0,"size":20,"query":{"bool":{"must":[{"simple_query_string":{"query":"\"events_ldi*\"","fields":["*^1.0"],"flags":-1,"default_operator":"or","lenient":true,"analyze_wildcard":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"fuzzy_transpositions":true,"boost":1.0}}],"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"index-pattern","boost":1.0}}},{"term":{"namespace":{"value":"events_ldi","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"_source":{"includes":["index-pattern.title","index-pattern.fields","index-pattern.timeFieldName","namespace","type","references","migrationVersion","updated_at","title","fields","timeFieldName"],"excludes":[]},"track_total_hits":2147483647}]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605819 -                              transport 1673435322946 11:08:42 272.1ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":48200,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605820 kAuGDJkjT0y0mmS0egOdbg:3605819 direct    1673435322946 11:08:42 272.1ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605823 -                              transport 1673435322956 11:08:42 262.7ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":800,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605824 kAuGDJkjT0y0mmS0egOdbg:3605823 direct    1673435322956 11:08:42 263.1ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605825 -                              transport 1673435322957 11:08:42 261.7ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":76200,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605826 kAuGDJkjT0y0mmS0egOdbg:3605825 direct    1673435322957 11:08:42 261.7ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605827 -                              transport 1673435322969 11:08:42 249.4ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":6000,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605828 kAuGDJkjT0y0mmS0egOdbg:3605827 direct    1673435322969 11:08:42 249.3ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605850 -                              transport 1673435322988 11:08:42 230.4ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":51700,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605851 kAuGDJkjT0y0mmS0egOdbg:3605850 direct    1673435322988 11:08:42 230.3ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605852 -                              transport 1673435322998 11:08:42 220.9ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":5200,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605853 kAuGDJkjT0y0mmS0egOdbg:3605852 direct    1673435322998 11:08:42 220.9ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]
indices:data/read/search              kAuGDJkjT0y0mmS0egOdbg:3605854 -                              transport 1673435322999 11:08:42 219.7ms     SERVERIP SERVERNAME indices[.kibana], types[], search_type[QUERY_THEN_FETCH], source[{"from":7300,"size":100,"query":{"bool":{"filter":[{"bool":{"should":[{"bool":{"must":[{"term":{"type":{"value":"application_usage_transactional","boost":1.0}}}],"must_not":[{"exists":{"field":"namespace","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"minimum_should_match":"1","boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"seq_no_primary_term":true,"track_total_hits":2147483647}]
indices:data/read/search[phase/query] kAuGDJkjT0y0mmS0egOdbg:3605855 kAuGDJkjT0y0mmS0egOdbg:3605854 direct    1673435322999 11:08:42 219.6ms     SERVERIP SERVERNAME shardId[[.kibana_1][0]]

Anyone have an idea for a solution please?

Thanks

11

If you stop Kibana the load in your machine goes down?

Do you have monitoring enabled in this Kibana?

Also, what type of disk are you using? You didn't say.

Not sure what those tasks mean, maybe someone from Elastic can give more context, but it seems that you have a lot of searchs tasks on the .kibana indice.

Do you have APM? I would assume that something like application_usage_transactional is related to APM.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.