Kibana url not working. FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead

SUNA · October 19, 2022, 3:44pm

Hi Team,

Kindly help here how can i fix this. ?

Previously, ELK failed due to an error "java.lang. IllegalStateException: A node cannot be upgraded directly from version [7.10.2] to version [8.4.3], it must first be upgraded to version [7.17.0] "Following that, we reinstalled Elasticsearch with version 7.17.0. However, the Kibana version remains 8.4.3.

Is it necessary to have both versions (elasticsearch and kibana)?

I restarted Elasticsearch, kibana, and logstash, but my kibana url is still down.
Kibana keeps failing, and the following errors have been observed.

Status of ufw: inactive

The server port status is shown below.

I could see 5601 is not open also
root@elk:~# netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 632/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 846/sshd: /usr/sbin
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 571359/java
tcp6 0 0 :::9200 :::* LISTEN 472296/java
tcp6 0 0 :::9300 :::* LISTEN 472296/java
tcp6 0 0 :::22 :::* LISTEN 846/sshd: /usr/sbin

Elastic search logs

root@elk:/var/log/elasticsearch# tail -f wakefit.log
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:349) [x-pack-ilm-7.17.0.jar:7.17.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186) [x-pack-core-7.17.0.jar:7.17.0]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220) [x-pack-core-7.17.0.jar:7.17.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2022-10-19T21:07:53,794][INFO ][o.e.x.i.IndexLifecycleRunner] [node-1] policy [apm-rollover-30-days] for index [apm-7.10.2-span] on an error step due to a transient error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [194]

Kibana service logs

root@elk:/var/log/elasticsearch# sudo systemctl status kibana

● kibana.service - Kibana

Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)

Active: failed (Result: exit-code) since Wed 2022-10-19 15:53:49 IST; 5h 16min ago

Process: 488438 ExecStart=/usr/share/kibana/bin/kibana (code=exited, status=78)

Main PID: 488438 (code=exited, status=78)

Oct 19 15:53:49 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 165.

Oct 19 15:53:49 elk systemd[1]: Stopped Kibana.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Start request repeated too quickly.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: Failed to start Kibana.

root@elk:/var/log/elasticsearch# sudo journalctl -fu kibana.service

-- Logs begin at Tue 2022-10-11 09:15:11 IST. --

Oct 19 15:53:46 elk kibana[488438]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)

Oct 19 15:53:46 elk kibana[488438]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)

Oct 19 15:53:46 elk kibana[488438]: FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: Service accounts | Elasticsearch Guide [8.0] | Elastic

Oct 19 15:53:46 elk systemd[1]: kibana.service: Main process exited, code=exited, status=78/CONFIG

Oct 19 15:53:46 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 165.

Oct 19 15:53:49 elk systemd[1]: Stopped Kibana.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Start request repeated too quickly.

Oct 19 15:53:49 elk systemd[1]: kibana.service: Failed with result 'exit-code'.

Oct 19 15:53:49 elk systemd[1]: Failed to start Kibana.

SUNA · October 19, 2022, 4:03pm

Could you please help how to troubleshoot this ? Kindly let us know if required any other information.

swchandu · October 19, 2022, 4:32pm

Kibana might be using "elastic" user ID. If this is not changed, try to replace with kibana_system(or other).

SUNA · October 19, 2022, 4:58pm

@swchandu kindly review the kibana.yml file below and help me to fix. Elastic is the user name I use. It was working fine for me.

server.port: 5601
server.host: "127.0.0.1"
server.name: "elk-server"
elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "RTMRpVBdtJwPhHR9"

swchandu · October 20, 2022, 6:00am

You would need to use another ID and not elastic

SUNA · October 20, 2022, 6:32am

I don't have any additional accounts. Please advise on how to accomplish this.

Also, since my kibana version is 8.4.3, can we upgrade my elasticsearch version from 7.17 to 8.4.3?

I believe kibana and elasticsearch should be the same version.

warkolm · October 23, 2022, 11:23pm

Welcome to our community! I wanted to point a few things if you don't mind!

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

You need to, yes. You can run different minor versions for a short term, but Elasticsearch 7.X is not (long term) compatible with Kibana 8.X. Support Matrix | Elastic shows you what is compatible.

As it mentions, you cannot use the elastic superuser as this is a reserved user.
See Security APIs | Elasticsearch Guide [8.0] | Elastic for how to create new users.

SUNA · October 24, 2022, 4:56am

Sure @warkolm

Currently i have restarored the sanpshot and debugging the issue. will reach you if any help required. thanks for your update.

SUNA · October 25, 2022, 7:28am

@warkolm

I have restored the snapshot. trying to debug the issue.
I have observed elasticsearch service failed. i'm unable to find the error logs and looks not updating the logs. "journalctl -xe" not showing any entries.

Rest of the services (kibana) running.

Could you help to debug this. My elastic and kibana installed in same server.

Elastic search logs

root@elk:~# tail -f /var/log/elasticsearch/wakefit.log 
	at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:275) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:212) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:272) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.node.Node.<init>(Node.java:362) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.node.Node.<init>(Node.java:289) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.10.2.jar:7.10.2]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.10.2.jar:7.10.2]
	... 6 more

ELK server port status

root@elk:~# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      636/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      842/sshd: /usr/sbin 
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      30788/node          
tcp6       0      0 :::5044                 :::*                    LISTEN      799/java            
tcp6       0      0 :::22                   :::*                    LISTEN      842/sshd: /usr/sbin 
tcp6       0      0 127.0.0.1:9600          :::*                    LISTEN      799/java            
root@elk:~#

Elasticsearch config file

root@elk:~# cat /etc/elasticsearch/elasticsearch.yml
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: wakefit
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /var/lib/elasticsearch
path.data: /mnt/elk_volume_1/elasticsearch

#path.data:
#    - /var/lib/elasticsearch
#    - /mnt/elk_volume_1/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 0.0.0.0
network.host: 127.0.0.1
#network.host: 139.59.60.240
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
discovery.type: single-node
#action.destructive_requires_name: true
xpack.security.enabled: true
http.cors.enabled:  true
#discovery.type: single-node
#xpack.security.transport.ssl.enabled: true
root@elk:~#

warkolm · October 26, 2022, 4:07am

Can you share more of your Elasticsearch logs please.

SUNA · October 26, 2022, 4:31am

@swchandu @warkolm Thanks a lot for your help.
above issue is fixed after changing service account access.

I have restored the snapshot .Will it be possible to restore dashboards as previous.?

SUNA · November 17, 2022, 3:25pm

@warkolm @leandrojmp

Could you please help to troubleshoot this issue. once i opened kibana url i could see events only.

logs are not fetching from the server
Please check below logs in dashboard

6,332 hits
Nov 17, 2022 @ 20:12:13.704 - Nov 17, 2022 @ 20:27:13.704

Auto
Documents
Time	_source
	Nov 17, 2022 @ 20:27:06.699	@timestamp:Nov 17, 2022 @ 20:27:06.699 event.duration:52.1 event.dataset:system.process event.module:system metricset.period:10,000 metricset.name:process service.type:system ecs.version:1.6.0 agent.id:51a42687-82dc-4592-82ea-b56b40474214 agent.name:app_server_69 agent.type:metricbeat agent.version:7.10.0 agent.hostname:ip-10-0-1-69 agent.ephemeral_id:e717c31f-ca32-45e7-9d95-3a057871d4bb cloud.image.id:ami-01c84af8788488ad0 cloud.instance.id:i-0112ac3ffb6cd3875 cloud.provider:aws cloud.machine.type:c5.4xlarge cloud.region:ap-south-1 cloud.availability_zone:ap-south-1a cloud.account.id:034177426869 process.args:/var/www/api.wakefit.co/venv/bin/python3.6, /var/www/api.wakefit.co/venv/bin/gunicorn, --access-logfile, /var/log/gunicorn/access.log, --error-logfile, /var/log/gunicorn/error.log, --timeout, 240, --log-level, debug, --workers, 50, --enable-stdio-inheritance, --capture-output, --bind, unix:/run/gunicorn.sock, wakefit.wsgi:application process.name:gunicorn process.pid:7103 process.ppid:22127 process.pgid:22127 process.working_directory:/var/www/api.wakefit.co process.executable:/usr/bin/python3.6 user.name:ubuntu system.process.fd.open:24 system.process.fd.limit.hard:4,096 system.process.fd.limit.soft:1,024 system.process.state:sleeping system.process.cpu.total.value:38,180 system.process.cpu.total.pct:8.4% system.process.cpu.total.norm.pct:0.53% system.process.cpu.start_time:Nov 17, 2022 @ 19:39:12.000 system.process.cmdline:/var/www/api.wakefit.co/venv/bin/python3.6 /var/www/api.wakefit.co/venv/bin/gunicorn --access-logfile /var/log/gunicorn/access.log --error-logfile /var/log/gunicorn/error.log --timeout 240 --log-level debug --workers 50 --enable-stdio-inheritance --capture-output --bind unix:/run/gunicorn.sock wakefit.wsgi:application system.process.cgroup.memory.id:gunicorn.service system.process.cgroup.memory.path:/system.slice/gunicorn.service system.process.cgroup.memory.mem.failures:0 system.process.cgroup.memory.mem.limit.bytes:8EB system.process.cgroup.memory.mem.usage.max.bytes:25.5GB system.process.cgroup.memory.mem.usage.bytes:20.3GB system.process.cgroup.memory.memsw.failures:0 system.process.cgroup.memory.memsw.limit.bytes:0B system.process.cgroup.memory.memsw.usage.bytes:0B system.process.cgroup.memory.memsw.usage.max.bytes:0B system.process.cgroup.memory.kmem.failures:0 system.process.cgroup.memory.kmem.limit.bytes:8EB system.process.cgroup.memory.kmem.usage.bytes:161.5MB system.process.cgroup.memory.kmem.usage.max.bytes:185.8MB system.process.cgroup.memory.kmem_tcp.usage.bytes:0B system.process.cgroup.memory.kmem_tcp.usage.max.bytes:0B system.process.cgroup.memory.kmem_tcp.failures:0 system.process.cgroup.memory.kmem_tcp.limit.bytes:8EB system.process.cgroup.memory.stats.inactive_file.bytes:599.1MB system.process.cgroup.memory.stats.pages_out:22,834,561 system.process.cgroup.memory.stats.pages_in:28,118,673 system.process.cgroup.memory.stats.swap.bytes:0B system.process.cgroup.memory.stats.cache.bytes:1.4GB system.process.cgroup.memory.stats.page_faults:41,171,856 system.process.cgroup.memory.stats.hierarchical_memsw_limit.bytes:0B system.process.cgroup.memory.stats.inactive_anon.bytes:0B system.process.cgroup.memory.stats.active_anon.bytes:18.8GB system.process.cgroup.memory.stats.mapped_file.bytes:924KB system.process.cgroup.memory.stats.rss.bytes:18.8GB system.process.cgroup.memory.stats.active_file.bytes:819.8MB system.process.cgroup.memory.stats.unevictable.bytes:0B system.process.cgroup.memory.stats.hierarchical_memory_limit.bytes:8EB system.process.cgroup.memory.stats.major_page_faults:198 system.process.cgroup.memory.stats.rss_huge.bytes:0B system.process.cgroup.blkio.path:/system.slice/gunicorn.service system.process.cgroup.blkio.total.bytes:65.6MB system.process.cgroup.blkio.total.ios:3,963 system.process.cgroup.blkio.id:gunicorn.service system.process.cgroup.id:gunicorn.service system.process.cgroup.path:/system.slice/gunicorn.service system.process.cgroup.cpu.cfs.quota.us:0 system.process.cgroup.cpu.cfs.shares:1,024 system.process.cgroup.cpu.cfs.period.us:100,000 system.process.cgroup.cpu.rt.runtime.us:0 system.process.cgroup.cpu.rt.period.us:0 system.process.cgroup.cpu.stats.throttled.ns:0 system.process.cgroup.cpu.stats.throttled.periods:0 system.process.cgroup.cpu.stats.periods:0 system.process.cgroup.cpu.id:gunicorn.service system.process.cgroup.cpu.path:/system.slice/gunicorn.service system.process.cgroup.cpuacct.path:/system.slice/gunicorn.service system.process.cgroup.cpuacct.total.ns:37,288,027,409,844 system.process.cgroup.cpuacct.stats.system.ns:1,621,130,000,000 system.process.cgroup.cpuacct.stats.user.ns:35,494,060,000,000 system.process.cgroup.cpuacct.percpu.1:1,767,683,453,400 system.process.cgroup.cpuacct.percpu.2:2,426,385,434,573 system.process.cgroup.cpuacct.percpu.3:2,360,895,309,350 system.process.cgroup.cpuacct.percpu.4:2,398,474,846,273 system.process.cgroup.cpuacct.percpu.5:2,433,103,281,476 system.process.cgroup.cpuacct.percpu.6:2,396,369,634,670 system.process.cgroup.cpuacct.percpu.7:1,573,104,978,704 system.process.cgroup.cpuacct.percpu.8:2,544,746,992,937 system.process.cgroup.cpuacct.percpu.9:2,418,113,299,650 system.process.cgroup.cpuacct.percpu.10:2,707,133,024,542 system.process.cgroup.cpuacct.percpu.11:2,478,944,970,407 system.process.cgroup.cpuacct.percpu.12:2,547,332,471,461 system.process.cgroup.cpuacct.percpu.13:2,314,417,078,545 system.process.cgroup.cpuacct.percpu.14:2,452,692,969,286 system.process.cgroup.cpuacct.percpu.15:1,621,403,958,028 system.process.cgroup.cpuacct.percpu.16:2,847,225,706,542 system.process.cgroup.cpuacct.id:gunicorn.service system.process.memory.size:1.1GB system.process.memory.rss.pct:0.75% system.process.memory.rss.bytes:235.6MB system.process.memory.share:39.8MB host.id:dcadcb950e3949fcb70593dade766397 host.containerized:false host.ip:10.0.1.69, fe80::a9:3bff:fec7:382 host.mac:02:a9:3b:c7:03:82 host.name:app_server_69 host.hostname:ip-10-0-1-69 host.architecture:x86_64 host.os.codename:bionic host.os.platform:ubuntu host.os.version:18.04.4 LTS (Bionic Beaver) host.os.family:debian host.os.name:Ubuntu host.os.kernel:5.4.0-1088-aws _id:ONQXhoQBX1mpuHxX9Npe _type:_doc _index:metricbeat-7.10.0-2022.10.25-000001 _score: -
	Nov 17, 2022 @ 20:27:06.699	@timestamp:Nov 17, 2022 @ 20:27:06.699 event.dataset:system.process event.module:system event.duration:52.1 metricset.name:process metricset.period:10,000 host.mac:02:a9:3b:c7:03:82 host.name:app_server_69 host.hostname:ip-10-0-1-69 host.architecture:x86_64 host.os.family:debian host.os.name:Ubuntu host.os.kernel:5.4.0-1088-aws host.os.codename:bionic host.os.platform:ubuntu host.os.version:18.04.4 LTS (Bionic Beaver) host.id:dcadcb950e3949fcb70593dade766397 host.containerized:false host.ip:10.0.1.69, fe80::a9:3bff:fec7:382

warkolm · November 18, 2022, 12:04am

Please start a new topic for your other question

SUNA · November 18, 2022, 12:28am

Sure @warkolm Thanks.

system · December 16, 2022, 12:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana error [ FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden.] Kibana elastic-stack-security	7	19640	March 23, 2022
Kibana error: not allowed to use elastic user Kibana elastic-stack-security	2	2938	June 7, 2022
Elastic search(8.2) and Kibana Authentication Error Kibana	11	4323	July 24, 2022
Cannot upgrade to Elasticsearch 8.12.1 directly from 7.12.0 Elasticsearch	16	908	April 4, 2024
After Update from Kibana 7.x to Kibana 8.9 =>security_exception: unable to authenticate user [kibana_system] for REST request [/_cluster/settings?include_defaults=true&f Kibana elastic-stack-security	7	1174	September 18, 2023

Kibana url not working. FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead

Elastic search logs

Kibana service logs

Related Topics