Kibana url not working. FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead

SUNA · October 19, 2022, 3:44pm

Hi Team,

Kindly help here how can i fix this. ?

Previously, ELK failed due to an error "java.lang. IllegalStateException: A node cannot be upgraded directly from version [7.10.2] to version [8.4.3], it must first be upgraded to version [7.17.0] "Following that, we reinstalled Elasticsearch with version 7.17.0. However, the Kibana version remains 8.4.3.

Is it necessary to have both versions (elasticsearch and kibana)?

I restarted Elasticsearch, kibana, and logstash, but my kibana url is still down.
Kibana keeps failing, and the following errors have been observed.

Status of ufw: inactive

The server port status is shown below.

I could see 5601 is not open also
root@elk:~# netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 632/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 846/sshd: /usr/sbin
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 571359/java
tcp6 0 0 :::9200 :::* LISTEN 472296/java
tcp6 0 0 :::9300 :::* LISTEN 472296/java
tcp6 0 0 :::22 :::* LISTEN 846/sshd: /usr/sbin

Elastic search logs

root@elk:/var/log/elasticsearch# tail -f wakefit.log
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) [x-pack-ilm-7.17.0.jar:7.17.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) [x-pack-core-7.17.0.jar:7.17.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) [x-pack-core-7.17.0.jar:7.17.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-10-19T21:07:53,794][INFO ][o.e.x.i.IndexLifecycleRunner] [node-1] policy [apm-rollover-30-days] for index [apm-7.10.2-span] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [194]

Kibana service logs

root@elk:/var/log/elasticsearch# sudo systemctl status kibana

● kibana.service - Kibana

Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Wed 2022-10-19 15:53:49 IST; 5h 16min ago

Process: 488438 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=78)

Main PID: 488438 (code=exited, status=78)

Oct 19 15:53:49 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 165.

Oct 19 15:53:49 elk systemd[1]: Stopped Kibana.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Start request repeated too quickly.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: Failed to start Kibana.

root@elk:/var/log/elasticsearch# sudo journalctl -fu kibana.service

-- Logs begin at Tue 2022-10-11 09:15:11 IST. --

Oct 19 15:53:46 elk kibana[488438]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)

Oct 19 15:53:46 elk kibana[488438]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)

Oct 19 15:53:46 elk kibana[488438]: FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: Service accounts | Elasticsearch Guide [8.0] | Elastic

Oct 19 15:53:46 elk systemd[1]: kibana.service: Main process exited, code=exited, status=78/CONFIG

Oct 19 15:53:46 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 165.

Oct 19 15:53:49 elk systemd[1]: Stopped Kibana.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Start request repeated too quickly.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: Failed to start Kibana.

SUNA · October 19, 2022, 4:03pm

Could you please help how to troubleshoot this ? Kindly let us know if required any other information.

swchandu · October 19, 2022, 4:32pm

Kibana might be using "elastic" user ID. If this is not changed, try to replace with kibana_system(or other).

SUNA · October 19, 2022, 4:58pm

@swchandu kindly review the kibana.yml file below and help me to fix. Elastic is the user name I use. It was working fine for me.

server.port: 5601
server.host: "127.0.0.1"
server.name: "elk-server"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "RTMRpVBdtJwPhHR9"

swchandu · October 20, 2022, 6:00am

You would need to use another ID and not elastic

SUNA · October 20, 2022, 6:32am

I don't have any additional accounts. Please advise on how to accomplish this.

Also, since my kibana version is 8.4.3, can we upgrade my elasticsearch version from 7.17 to 8.4.3?

I believe kibana and elasticsearch should be the same version.

warkolm · October 23, 2022, 11:23pm

Welcome to our community! I wanted to point a few things if you don't mind!

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

You need to, yes. You can run different minor versions for a short term, but Elasticsearch 7.X is not (long term) compatible with Kibana 8.X. Support Matrix | Elastic shows you what is compatible.

As it mentions, you cannot use the elastic superuser as this is a reserved user.
See Security APIs | Elasticsearch Guide [8.0] | Elastic for how to create new users.

SUNA · October 24, 2022, 4:56am

Sure @warkolm

Currently i have restarored the sanpshot and debugging the issue. will reach you if any help required. thanks for your update.

SUNA · October 25, 2022, 7:28am

@warkolm

I have restored the snapshot. trying to debug the issue.
I have observed elasticsearch service failed. i'm unable to find the error logs and looks not updating the logs. "journalctl -xe" not showing any entries.

Rest of the services (kibana) running.

Could you help to debug this. My elastic and kibana installed in same server.

Elastic search logs

root@elk:~# tail -f /var/log/elasticsearch/wakefit.log 
	at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.node.Node.<init>(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
	... 6 more

ELK server port status

root@elk:~# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      636/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      842/sshd: /usr/sbin 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      30788/node          
tcp6       0      0 :::5044                 :::*                    LISTEN      799/java            
tcp6       0      0 :::22                   :::*                    LISTEN      842/sshd: /usr/sbin 
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      799/java            
root@elk:~#

Elasticsearch config file

root@elk:~# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: wakefit
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /var/lib/elasticsearch
path.data: /mnt/elk_volume_1/elasticsearch

#path.data:
#    - /var/lib/elasticsearch
#    - /mnt/elk_volume_1/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 0.0.0.0
network.host: 127.0.0.1
#network.host: 139.59.60.240
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
discovery.type: single-node
#action.destructive_requires_name: true
xpack.security.enabled: true
http.cors.enabled:  true
#discovery.type: single-node
#xpack.security.transport.ssl.enabled: true
root@elk:~#

warkolm · October 26, 2022, 4:07am

Can you share more of your Elasticsearch logs please.

SUNA · October 26, 2022, 4:31am

@swchandu @warkolm Thanks a lot for your help.
above issue is fixed after changing service account access.

I have restored the snapshot .Will it be possible to restore dashboards as previous.?

SUNA · November 17, 2022, 3:25pm

@warkolm @leandrojmp

Could you please help to troubleshoot this issue. once i opened kibana url i could see events only.

logs are not fetching from the server
Please check below logs in dashboard

6,332 hits
Nov 17, 2022 @ 20:12:13.704 - Nov 17, 2022 @ 20:27:13.704

Auto
Documents
Time	_source
	Nov 17, 2022 @ 20:27:06.699	@timestamp:Nov 17, 2022 @ 20:27:06.699 event.duration:52.1 event.dataset:system.process event.module:system metricset.period:10,000 metricset.name:process service.type:system ecs.version:1.6.0 agent.id:51a42687-82dc-4592-82ea-b56b40474214 agent.name:app_server_69 agent.type:metricbeat agent.version:7.10.0 agent.hostname:ip-10-0-1-69 agent.ephemeral_id:e717c31f-ca32-45e7-9d95-3a057871d4bb cloud.image.id:ami-01c84af8788488ad0 cloud.instance.id:i-0112ac3ffb6cd3875 cloud.provider:aws cloud.machine.type:c5.4xlarge cloud.region:ap-south-1 cloud.availability_zone:ap-south-1a cloud.account.id:034177426869 process.args:/var/www/api.wakefit.co/venv/bin/python3.6, /var/www/api.wakefit.co/venv/bin/gunicorn, --access-logfile, /var/log/gunicorn/access.log, --error-logfile, /var/log/gunicorn/error.log, --timeout, 240, --log-level, debug, --workers, 50, --enable-stdio-inheritance, --capture-output, --bind, unix:/run/gunicorn.sock, wakefit.wsgi:application process.name:gunicorn process.pid:7103 process.ppid:22127 process.pgid:22127 process.working_directory:/var/www/api.wakefit.co process.executable:/usr/bin/python3.6 user.name:ubuntu system.process.fd.open:24 system.process.fd.limit.hard:4,096 system.process.fd.limit.soft:1,024 system.process.state:sleeping system.process.cpu.total.value:38,180 system.process.cpu.total.pct:8.4% system.process.cpu.total.norm.pct:0.53% system.process.cpu.start_time:Nov 17, 2022 @ 19:39:12.000 system.process.cmdline:/var/www/api.wakefit.co/venv/bin/python3.6 /var/www/api.wakefit.co/venv/bin/gunicorn --access-logfile /var/log/gunicorn/access.log --error-logfile /var/log/gunicorn/error.log --timeout 240 --log-level debug --workers 50 --enable-stdio-inheritance --capture-output --bind unix:/run/gunicorn.sock wakefit.wsgi:application system.process.cgroup.memory.id:gunicorn.service system.process.cgroup.memory.path:/system.slice/gunicorn.service system.process.cgroup.memory.mem.failures:0 system.process.cgroup.memory.mem.limit.bytes:8EB system.process.cgroup.memory.mem.usage.max.bytes:25.5GB system.process.cgroup.memory.mem.usage.bytes:20.3GB system.process.cgroup.memory.memsw.failures:0 system.process.cgroup.memory.memsw.limit.bytes:0B system.process.cgroup.memory.memsw.usage.bytes:0B system.process.cgroup.memory.memsw.usage.max.bytes:0B system.process.cgroup.memory.kmem.failures:0 system.process.cgroup.memory.kmem.limit.bytes:8EB system.process.cgroup.memory.kmem.usage.bytes:161.5MB system.process.cgroup.memory.kmem.usage.max.bytes:185.8MB system.process.cgroup.memory.kmem_tcp.usage.bytes:0B system.process.cgroup.memory.kmem_tcp.usage.max.bytes:0B system.process.cgroup.memory.kmem_tcp.failures:0 system.process.cgroup.memory.kmem_tcp.limit.bytes:8EB system.process.cgroup.memory.stats.inactive_file.bytes:599.1MB system.process.cgroup.memory.stats.pages_out:22,834,561 system.process.cgroup.memory.stats.pages_in:28,118,673 system.process.cgroup.memory.stats.swap.bytes:0B system.process.cgroup.memory.stats.cache.bytes:1.4GB system.process.cgroup.memory.stats.page_faults:41,171,856 system.process.cgroup.memory.stats.hierarchical_memsw_limit.bytes:0B system.process.cgroup.memory.stats.inactive_anon.bytes:0B system.process.cgroup.memory.stats.active_anon.bytes:18.8GB system.process.cgroup.memory.stats.mapped_file.bytes:924KB system.process.cgroup.memory.stats.rss.bytes:18.8GB system.process.cgroup.memory.stats.active_file.bytes:819.8MB system.process.cgroup.memory.stats.unevictable.bytes:0B system.process.cgroup.memory.stats.hierarchical_memory_limit.bytes:8EB system.process.cgroup.memory.stats.major_page_faults:198 system.process.cgroup.memory.stats.rss_huge.bytes:0B system.process.cgroup.blkio.path:/system.slice/gunicorn.service system.process.cgroup.blkio.total.bytes:65.6MB system.process.cgroup.blkio.total.ios:3,963 system.process.cgroup.blkio.id:gunicorn.service system.process.cgroup.id:gunicorn.service system.process.cgroup.path:/system.slice/gunicorn.service system.process.cgroup.cpu.cfs.quota.us:0 system.process.cgroup.cpu.cfs.shares:1,024 system.process.cgroup.cpu.cfs.period.us:100,000 system.process.cgroup.cpu.rt.runtime.us:0 system.process.cgroup.cpu.rt.period.us:0 system.process.cgroup.cpu.stats.throttled.ns:0 system.process.cgroup.cpu.stats.throttled.periods:0 system.process.cgroup.cpu.stats.periods:0 system.process.cgroup.cpu.id:gunicorn.service system.process.cgroup.cpu.path:/system.slice/gunicorn.service system.process.cgroup.cpuacct.path:/system.slice/gunicorn.service system.process.cgroup.cpuacct.total.ns:37,288,027,409,844 system.process.cgroup.cpuacct.stats.system.ns:1,621,130,000,000 system.process.cgroup.cpuacct.stats.user.ns:35,494,060,000,000 system.process.cgroup.cpuacct.percpu.1:1,767,683,453,400 system.process.cgroup.cpuacct.percpu.2:2,426,385,434,573 system.process.cgroup.cpuacct.percpu.3:2,360,895,309,350 system.process.cgroup.cpuacct.percpu.4:2,398,474,846,273 system.process.cgroup.cpuacct.percpu.5:2,433,103,281,476 system.process.cgroup.cpuacct.percpu.6:2,396,369,634,670 system.process.cgroup.cpuacct.percpu.7:1,573,104,978,704 system.process.cgroup.cpuacct.percpu.8:2,544,746,992,937 system.process.cgroup.cpuacct.percpu.9:2,418,113,299,650 system.process.cgroup.cpuacct.percpu.10:2,707,133,024,542 system.process.cgroup.cpuacct.percpu.11:2,478,944,970,407 system.process.cgroup.cpuacct.percpu.12:2,547,332,471,461 system.process.cgroup.cpuacct.percpu.13:2,314,417,078,545 system.process.cgroup.cpuacct.percpu.14:2,452,692,969,286 system.process.cgroup.cpuacct.percpu.15:1,621,403,958,028 system.process.cgroup.cpuacct.percpu.16:2,847,225,706,542 system.process.cgroup.cpuacct.id:gunicorn.service system.process.memory.size:1.1GB system.process.memory.rss.pct:0.75% system.process.memory.rss.bytes:235.6MB system.process.memory.share:39.8MB host.id:dcadcb950e3949fcb70593dade766397 host.containerized:false host.ip:10.0.1.69, fe80::a9:3bff:fec7:382 host.mac:02:a9:3b:c7:03:82 host.name:app_server_69 host.hostname:ip-10-0-1-69 host.architecture:x86_64 host.os.codename:bionic host.os.platform:ubuntu host.os.version:18.04.4 LTS (Bionic Beaver) host.os.family:debian host.os.name:Ubuntu host.os.kernel:5.4.0-1088-aws _id:ONQXhoQBX1mpuHxX9Npe _type:_doc _index:metricbeat-7.10.0-2022.10.25-000001 _score: -
	Nov 17, 2022 @ 20:27:06.699	@timestamp:Nov 17, 2022 @ 20:27:06.699 event.dataset:system.process event.module:system event.duration:52.1 metricset.name:process metricset.period:10,000 host.mac:02:a9:3b:c7:03:82 host.name:app_server_69 host.hostname:ip-10-0-1-69 host.architecture:x86_64 host.os.family:debian host.os.name:Ubuntu host.os.kernel:5.4.0-1088-aws host.os.codename:bionic host.os.platform:ubuntu host.os.version:18.04.4 LTS (Bionic Beaver) host.id:dcadcb950e3949fcb70593dade766397 host.containerized:false host.ip:10.0.1.69, fe80::a9:3bff:fec7:382

warkolm · November 18, 2022, 12:04am

Please start a new topic for your other question

SUNA · November 18, 2022, 12:28am

Sure @warkolm Thanks.

system · December 16, 2022, 12:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.