X-Pack 5.6.6 - heavy cpu load

Elastic Stack Elasticsearch
1

Hi,

I upgrade to Elastic 5.6.6 and also x-pack to 5.6.6 relase.

Now I have heavy cpu load .

Log of elasticsearch:

[2018-02-01T10:30:38,368][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [H12Ebd5] collector [cluster_stats] failed to collect data
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:130) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.6.6.jar:5.6.6]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[2018-02-01T10:30:42,595][WARN ][r.suppressed ] path: /.reporting-/esqueue/_search, params: {index=.reporting-, type=esqueue, version=true}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:130) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:107) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:205) ~[elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:184) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.6.jar:5.6.6]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.6.jar:5.6.6]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
[2018-02-01T10:30:42,601][WARN ][o.e.m.j.JvmGcMonitorService] [H12Ebd5] [gc][767] overhead, spent [4s] collecting in the last [4.4s]
[2018-02-01T10:30:42,644][ERROR][o.e.x.w.i.s.ExecutableSimpleInput] [H12Ebd5] failed to execute [search] input for watch [rtBJGdfpQWm3raH3-ymgWQ_logstash_version_mismatch], reason [all shards failed]
[2018-02-01T10:30:42,657][ERROR][o.e.x.w.i.s.ExecutableSimpleInput] [H12Ebd5] failed to execute [search] input for watch [rtBJGdfpQWm3raH3-ymgWQ_kibana_version_mismatch], reason [all shards failed]
[2018-02-01T10:30:42,655][ERROR][o.e.x.w.i.s.ExecutableSimpleInput] [H12Ebd5] failed to execute [search] input for watch [rtBJGdfpQWm3raH3-ymgWQ_elasticsearch_cluster_status], reason [all shards failed]
[2018-02-01T10:30:42,644][ERROR][o.e.x.w.i.s.ExecutableSimpleInput] [H12Ebd5] failed to execute [search] input for watch [rtBJGdfpQWm3raH3-ymgWQ_elasticsearch_version_mismatch], reason [all shards failed]
[2018-02-01T10:30:42,904][WARN ][o.e.x.w.e.ExecutionService] [H12Ebd5] Failed to execute watch [rtBJGdfpQWm3raH3-ymgWQ_elasticsearch_cluster_status_7cd9ba59-6eef-4592-808b-6070446b47c7-2018-02-01T09:30:42.600Z]
[2018-02-01T10:30:42,889][WARN ][o.e.x.w.e.ExecutionService] [H12Ebd5] Failed to execute watch [rtBJGdfpQWm3raH3-ymgWQ_kibana_version_mismatch_e57072d5-29f2-4870-8c0f-a818c06e720c-2018-02-01T09:30:42.600Z]
[2018-02-01T10:30:42,785][WARN ][o.e.x.w.e.ExecutionService] [H12Ebd5] Failed to execute watch [rtBJGdfpQWm3raH3-ymgWQ_logstash_version_mismatch_0f7bac81-e91a-4edf-928a-7b2440fb418b-2018-02-01T09:30:42.600Z]
[2018-02-01T10:30:43,013][WARN ][o.e.x.w.e.ExecutionService] [H12Ebd5] Failed to execute watch [rtBJGdfpQWm3raH3-ymgWQ_elasticsearch_version_mismatch_f43522d7-aad8-4785-8713-62fa1a0a95b8-2018-02-01T09:30:42.600Z]

2 
my elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: elastic-test
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /Elastic/
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
# network.host: localhost
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
# xpack.monitoring.collection.cluster.stats.timeout: 10m
xpack.watcher.enabled: true


xpack.security.enabled: true

xpack:
  security:
    authc:
      realms:
        active_directory:
          type: active_directory
          order: 0
          domain_name: my.domain.dom
          url: ldap://dc.my.domain.dom:3268


xpack.notification.email:
    default_account: smtp_account
    account:
        smtp_account:
            profile: standard
            smtp:
                host: smtp.my.domain.dom
                user: elastic@my.domain-mail.com
                from: elastic-from@my.domain-mail.com


Regards`Preformatted text`
3

Can you please edit your post and format the config using the </> button or markdown style backticks to make it readable.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.