We have kibana 7.4 installed on two servers where elasticsearch is also installed. Kibana service is getting restarted after some time.
Can someone please help.
Here is the kibana config file -
cat /etc/kibana/kibana.yml
# Ansible managed
server.port: 8601
server.host: "0.0.0.0"
server.name: kibana_2
elasticsearch.requestTimeout: 30000
elasticsearch.hosts: ['http://es1_ip:5200', 'http://es2_ip:5200', 'http://es3_ip:5200']
elasticsearch.username: "elastic"
elasticsearch.password: "XXXXXX"
kibana service file -
cat /etc/systemd/system/kibana.service
[Unit]
Description=Kibana
StartLimitIntervalSec=30
StartLimitBurst=3
[Service]
Type=simple
User=kibana
Group=kibana
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/kibana
EnvironmentFile=-/etc/sysconfig/kibana
ExecStart=/usr/share/kibana/bin/kibana "-c /etc/kibana/kibana.yml"
Restart=always
WorkingDirectory=/
[Install]
WantedBy=multi-user.target
-rw-rw---- 1 root kibana 331 Apr 16 20:06 kibana.yml
Here is elasticsearch config -
cat /etc/elasticsearch/elasticsearch.yml
# Ansible managed
node.name: elasticsearch_1
path.data: /opt/elasticsearch
path.logs: /var/log/elasticsearch
http.port: 5200
transport.port: 5300
path.repo: /opt/elastic_snapshot
cluster.name: xxx
cluster.initial_master_nodes: ['es1_ip', 'es2_ip', es3_ip']
discovery.seed_hosts: ['es1_ip', 'es2_ip', es3_ip']
network.host: 0.0.0.0
http.host: 0.0.0.0
node.master: True
node.data: True
node.ingest: True
discovery.zen.minimum_master_nodes: 2
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elasticsearch_1.p12
xpack.security.transport.ssl.truststore.path: elasticsearch_1.p12
# xpack.security.http.ssl.verification_mode: certificate
# xpack.security.http.ssl.keystore.path: elasticsearch_1.p12
# xpack.security.http.ssl.truststore.path: elasticsearch_1.p12
xpack.monitoring.collection.enabled: true
xpack.security.enabled: true
both process are showing as running but kibana gets restarted.
ps -ef |grep kibana
root 99171 93611 0 20:26 pts/0 00:00:00 grep --color=auto kibana
kibana 99172 1 0 20:26 ? 00:00:00 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
ps -ef |grep elasticsearch
elastic+ 90792 1 3 Apr15 ? 00:42:43 /usr/share/elasticsearch/jdk/bin/java -Xms5288m -Xmx5288m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-9885684502051401189 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Xms5288m -Xmx5288m -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=2772434944 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
elastic+ 90948 90792 0 Apr15 ? 00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
root 99588 93611 0 20:28 pts/0 00:00:00 grep --color=auto elasticsearch
systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/elasticsearch.service.d
└─startup-timeout.conf
Active: active (running) since Thu 2021-04-15 21:13:31 IST; 22h ago
Docs: http://www.elastic.co
Main PID: 108327 (java)
CGroup: /system.slice/elasticsearch.service
├─108327 /usr/share/elasticsearch/jdk/bin/java -Xms5288m -Xmx5288m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch-4761301156106937349 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -Djava.locale.providers=COMPAT -Xms5288m -Xmx5288m -Dio.netty.allocator.type=pooled -XX:MaxDirectMemorySize=2772434944 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
└─108449 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Apr 15 21:12:15 systemd[1]: Starting Elasticsearch...
Apr 15 21:12:22 elasticsearch[108327]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Apr 15 21:13:31 systemd[1]: Started Elasticsearch.
Getting connection refused when checking kibana.
curl -I http://127.0.0.1:8601/status
HTTP/1.1 503 Service Unavailable
retry-after: 30
content-type: text/html; charset=utf-8
cache-control: no-cache
content-length: 30
Date: Fri, 16 Apr 2021 14:39:47 GMT
Connection: keep-alive
curl -s http://127.0.0.1:8601/api/status
Kibana server is not ready yet
curl -v -s http://127.0.0.1:8601/api/status | jsonpp
* About to connect() to 127.0.0.1 port 8601 (#0)
* Trying 127.0.0.1...
* Connection refused
* Failed connect to 127.0.0.1:8601; Connection refused
* Closing connection 0
Getting below error while checking the health of es.
curl -XGET 'http://localhost:5200/_cluster/health' -u elastic
Enter host password for user 'elastic':
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}