Elasticsearch will not start after we installed 7.9.2. When started it enters a failed state. I ran journalctl -xe it points to Unregistered Authentication Agent. But when I check the logs I see the following:
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:273) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:240) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationPermit(IndexShard.java:2563) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.shard.IndexShard.runUnderPrimaryPermit(IndexShard.java:2639) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.IndexService.sync(IndexService.java:799) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.IndexService.syncRetentionLeases(IndexService.java:782) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.IndexService.access$800(IndexService.java:100) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.index.IndexService$AsyncRetentionLeaseSyncTask.runInternal(IndexService.java:960) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.common.util.concurrent.AbstractAsyncTask.run(AbstractAsyncTask.java:144) ~[elasticsearch-6.8.7.jar:6.8.7]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.8.7.jar:6.8.7]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_232]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_232]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_232]
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:626) ~[elasticsearch-6.8.7.jar:6.8.7]
... 55 more
[2021-02-04T16:54:46,536][INFO ][o.e.n.Node ] [dev-es-01] stopped
[2021-02-04T16:54:46,537][INFO ][o.e.n.Node ] [dev-es-01] closing ...
[2021-02-04T16:54:46,554][INFO ][o.e.n.Node ] [dev-es-01] closed
Are those the GC logs not the regular logs like above... We need a look at the regular elastic logs either though the journal like above or /var/log/elasticsearch/elasticsearch.log
There should be some messages near the end that indicate the issues... that above shows nothing...
Regular logs should look like something like this...
[2021-02-25T16:00:29,498][INFO ][o.e.p.PluginsService ] [ceres] loaded module [aggs-matrix-stats]
[2021-02-25T16:00:29,498][INFO ][o.e.p.PluginsService ] [ceres] loaded module [analysis-common]
[2021-02-25T16:00:29,498][INFO ][o.e.p.PluginsService ] [ceres] loaded module [constant-keyword]
[2021-02-25T16:00:29,498][INFO ][o.e.p.PluginsService ] [ceres] loaded module [flattened]
[2021-02-25T16:00:29,498][INFO ][o.e.p.PluginsService ] [ceres] loaded module [frozen-indices]
[2021-02-25T16:00:43,881][INFO ][o.e.c.r.a.AllocationService] [ceres] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[metricbeat-7.10.2-2021.02.11-000001][0]]]).
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-00
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /elasticsearch/data
#
# Path to log files:
#
path.logs: /elasticsearch/log
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts:
- 1xx.xx.x.xxx:9200
- 1xx.xx.x.xxx:9200
- 1xx.xx.x.xxx:9200
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes:
- domain-00
- domain-01
- domain-02
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
action.destructive_requires_name: true
reindex.remote.whitelist: "Linuxvm:9202,Linuxvm2:443"
reindex.ssl.verification_mode : "none"
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.graph.enabled: true
xpack.watcher.enabled: true
xpack.monitoring.exporters.local_default2:
type: local
xpack.ssl.keystore.path: certs/elastic-certificates.p12
xpack.ssl.truststore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.http.ssl.enabled: true
xpack.security.audit.enabled: true
xpack:
security:
authc:
realms:
native1:
type: native
order: 0
active_directory:
type: active_directory
order: 1
domain_name:
url:
bind_dn:
load_balance:
type: "round_robin"
unmapped_groups_as_roles: true
file1:
type: file
order: 0
path.repo: /net/snaps
Feb 26 12:59:47 dev-es-00.autumnal.local systemd[1]: Starting Elasticsearch...
Feb 26 13:00:46 dev-es-00.autumnal.local systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Feb 26 13:00:46 dev-es-00.autumnal.local systemd[1]: Failed to start Elasticsearch.
Feb 26 13:00:46 dev-es-00.autumnal.local systemd[1]: Unit elasticsearch.service entered failed state.
Feb 26 13:00:46 dev-es-00.autumnal.local systemd[1]: elasticsearch.service failed.
Unit status.logs.service could not be found.
[root@node mwills@domain]# systemctl start elasticsearch || journalctl -xe
Job for elasticsearch.service failed because a fatal signal was delivered to the control process. See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
Journal file /var/log/journal/2b2dca37c19647b09cf1f366fc7406ea/user-815601170.journal is truncated, ignoring file.
-- Subject: Session 34524 has been terminated
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- Documentation: http://www.freedesktop.org/wiki/Software/systemd/multiseat
--
-- A session with the ID 34524 has been terminated.
Feb 26 14:46:30 node dbus[746]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatch
Feb 26 14:46:30 node dhclient[853]: bound to 172.29.1.179 -- renewal in 1610 seconds.
Feb 26 14:46:31 node systemd[1]: Starting Network Manager Script Dispatcher Service...
-- Subject: Unit NetworkManager-dispatcher.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit NetworkManager-dispatcher.service has begun starting up.
Feb 26 14:46:31 node systemd[1]: Started Session 34536 of user root.
-- Subject: Unit session-34536.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-34536.scope has finished starting up.
--
-- The start-up result is done.
Feb 26 14:46:31 node CROND[11723]: (root) CMD (/usr/bin/aws-kinesis-agent-babysit > /dev/null 2>&1)
Feb 26 14:46:31 node dbus[746]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Feb 26 14:46:31 node systemd[1]: Started Network Manager Script Dispatcher Service.
-- Subject: Unit NetworkManager-dispatcher.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit NetworkManager-dispatcher.service has finished starting up.
--
-- The start-up result is done.
Feb 26 14:46:31 node nm-dispatcher[11722]: req:1 'dhcp4-change' [eth0]: new request (4 scripts)
Feb 26 14:46:31 node nm-dispatcher[11722]: req:1 'dhcp4-change' [eth0]: start running ordered scripts...
lines 1980-2013/2013 (END)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.