Hello, Team.
We have an ELK server in our digital environment, as well as production servers in AWS.
We have recently encountered version dependency issues in ELK. As a result, I intend to restore ELK server from an old snapshot.
Please advise what changes I should make at the configuration level.
What sort of changes are you referring to or expecting to have to change?
Finallay I have to bring up my elastic server back to normal.
FYI.
Below are the versions I see in both elasticserach and kibana versions.
root@elk:/usr/share/elasticsearch# sudo ./bin/elasticsearch --version
Version: 8.4.3, Build: deb/42f05b9372a9a4a470db3b52817899b99a76ee73/2022-10-04T07:17:24.662462378Z, JVM: 18.0.2.1
Kibana : 8.4.3 version
sudo /usr/share/kibana/bin/kibana --version --allow-root
8.4.3
@warkolm Currently my elastic service failed with error "[2022-10-22T11:47:16,777][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started"
Kibana status and logs
root@elk:/usr/share/elasticsearch# sudo systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2022-10-20 12:58:21 IST; 1 day 23h ago
Main PID: 835605 (code=exited, status=78)
Oct 20 12:58:21 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 9.
Oct 20 12:58:21 elk systemd[1]: Stopped Kibana.
Oct 20 12:58:21 elk systemd[1]: kibana.service: Start request repeated too quickly.
Oct 20 12:58:21 elk systemd[1]: kibana.service: Failed with result 'exit-code'.
Oct 20 12:58:21 elk systemd[1]: Failed to start Kibana.
root@elk:/usr/share/elasticsearch# sudo journalctl -fu kibana.service
-- Logs begin at Thu 2022-10-13 16:41:01 IST. --
Oct 20 12:58:17 elk kibana[835605]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
Oct 20 12:58:17 elk kibana[835605]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
Oct 20 12:58:17 elk kibana[835605]: FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: Service accounts | Elasticsearch Guide [8.0] | Elastic
Elasticsearch service failed with below error
root@elk:/usr/share/elasticsearch# tail -f /var/log/elasticsearch/wakefit.log
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.3.jar:?]
... 8 more
[2022-10-22T11:47:16,561][INFO ][o.e.n.Node ] [node-1] stopping ...
[2022-10-22T11:47:16,733][INFO ][o.e.n.Node ] [node-1] stopped
[2022-10-22T11:47:16,733][INFO ][o.e.n.Node ] [node-1] closing ...
[2022-10-22T11:47:16,770][INFO ][o.e.n.Node ] [node-1] closed
[2022-10-22T11:47:16,777][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
Elasticalerts status and logs
root@elk:/usr/share/elasticsearch# systemctl status elastalert.service
● elastalert.service - elastalert
Loaded: loaded (/lib/systemd/system/elastalert.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2022-10-20 13:14:58 IST; 1 day 23h ago
Main PID: 840367 (elastalert)
Tasks: 4 (limit: 9507)
Memory: 47.8M
CGroup: /system.slice/elastalert.service
└─840367 /usr/bin/python3 /usr/local/bin/elastalert --verbose --config /opt/elastalert/config.yaml
Oct 22 12:25:44 elk elastalert[840367]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='143.110.240.197', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('
Oct 22 12:25:44 elk elastalert[840367]: During handling of the above exception, another exception occurred:
Oct 22 12:25:44 elk elastalert[840367]: Traceback (most recent call last):
Oct 22 12:25:44 elk elastalert[840367]: File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_requests.py", line 77, in perform_request
**Oct 22 12:25:44 elk elastalert[840367]: response = self.session.send(prepared_request, send_kwargs)
Oct 22 12:25:44 elk elastalert[840367]: File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 655, in send
**Oct 22 12:25:44 elk elastalert[840367]: r = adapter.send(request, kwargs)
Oct 22 12:25:44 elk elastalert[840367]: File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 516, in send
Oct 22 12:25:44 elk elastalert[840367]: raise ConnectionError(e, request=request)
Oct 22 12:25:44 elk elastalert[840367]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='143.110.240.197', port=9200): Max retries exceeded with url: / (Caused by NewConnectionError('
root@elk:/usr/share/elasticsearch# netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 632/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 846/sshd: /usr/sbin
tcp6 0 0 127.0.0.1:9600 :::* LISTEN 1606922/java
tcp6 0 0 :::22 :::* LISTEN 846/sshd: /usr/sbin
How to debug this issue. kindly help.
From which version did you upgrade to 8? Did you upgrade from 7.17.X to 8.X and solved any issue in the Upgrade Assistant page?
Your main issue here is that your Elasticsearch is not starting, and from the part of the log you shared, it is a know issue related to some deprecated mapping, but to confirm it you need to share more lines of the logs, the start of the last stack trace at least.
You basically have indices that have an unsupported mapping format, and since you shared some ElastAlert logs, I would assume that those indices are the elastalert_* indices, which uses the dateOptionalTime, which is deprecated and should've be changed to date_optional_time.
The problem is that there is some issue in the upgrade process that won't allow Elasticsearch to upgrade but will also not allow you to downgrade it, so you end with a non-working cluster.
There is a fix for it, but it seems to be planned to version 8.5, which was not released yet.
I don't think that there is a fix for your issue at the moment, you would need to build a new cluster and restore your data from snapshots, except the elastalert indices, you would need to reinstall it using the latest version of elastalert2 .
Thank you for your input.
Previously, my elasticsearch version worked perfectly with 7.10.2.
My elasticsearch and kibana suddenly stopped working. In elasticsearch logs, I discovered the following error.
java.lang.IllegalStateException: You cannot upgrade a node from version [7.10.2] to version [8.4.3] without first upgrading to version [7.17.0].
I was unable to start the elasticsearch service as a result of this error.
Later, I read on the elastic blog that the version should be the same for kibana and Elasticsearch.
I noticed that when I upgraded my kibana version 8.4.3, I first tried to upgrade ES version 7.17, and then I upgraded to 8.4.3.
Both the kibana and elastic versions are now the same.
Yes those logs are elastalert logs.
Elasticsearch logs :
root@elk:/usr/share/elasticsearch/bin# tail -f /var/log/elasticsearch/wakefit.log
at org.elasticsearch.index.mapper.MapperService.parseMapping(MapperService.java:370) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:347) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:337) ~[elasticsearch-8.4.3.jar:?]
at org.elasticsearch.cluster.metadata.IndexMetadataVerifier.checkMappingsCompatibility(IndexMetadataVerifier.java:197) ~[elasticsearch-8.4.3.jar:?]
... 8 more
[2022-10-22T19:21:00,307][INFO ][o.e.n.Node ] [node-1] stopping ...
[2022-10-22T19:21:00,405][INFO ][o.e.n.Node ] [node-1] stopped
[2022-10-22T19:21:00,405][INFO ][o.e.n.Node ] [node-1] closing ...
[2022-10-22T19:21:00,437][INFO ][o.e.n.Node ] [node-1] closed
[2022-10-22T19:21:00,441][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
Kibana service logs :
root@elk:/usr/share/elasticsearch/bin# sudo journalctl -fu kibana.service
-- Logs begin at Thu 2022-10-13 21:53:56 IST. --
Oct 22 12:36:53 elk kibana[1609796]: at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
Oct 22 12:36:53 elk kibana[1609796]: at Command. (/usr/share/kibana/src/cli/serve/serve.js:216:5)
Oct 22 12:36:53 elk kibana[1609796]: FATAL Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: Service accounts | Elasticsearch Guide [8.0] | Elastic
Oct 22 12:36:53 elk systemd[1]: kibana.service: Main process exited, code=exited, status=78/CONFIG
Oct 22 12:36:53 elk systemd[1]: kibana.service: Failed with result 'exit-code'.
Oct 22 12:36:56 elk systemd[1]: kibana.service: Scheduled restart job, restart counter is at 6.
Oct 22 12:36:56 elk systemd[1]: Stopped Kibana.
Oct 22 12:36:56 elk systemd[1]: kibana.service: Start request repeated too quickly.
Oct 22 12:36:56 elk systemd[1]: kibana.service: Failed with result 'exit-code'.
Oct 22 12:36:56 elk systemd[1]: Failed to start Kibana.
Sure, I'll look into it as you suggested.
I'd like to confirm whether I'll create new "clastic" ( service account) credentials credentilas after restoring the snapshot. or will old elastic credentials work just fine?
Please help me steps to create service account credentials
You need to ignore Kibana errors or any other tool error by now, your Elasticsearch is not working, so any tool that communicates with elasticsearch will not work and you can do nothing to fix it until your Elasticsearch is working.
Did you upgrade from 7.10.2 to 7.17and checked the upgrade assistant to solve any issue before upgrading to 8.4.3?
It is not clear what upgrade path you followed.
As I said before, you have a non-working cluster because of a mapping issue caused by the elastalert indices, there is no fix at the moment, the fix seems to be planned to arrive in 8.5, check the explanation in the previous answer.
You do not have many options to solve this as your elasticsearch will not start because of an issue that is still not solved in any released version.
If you have snapshot backups of your indices you will need to create an entire new cluster and restore the snapshot from your repository, keep in mind that these are snapshots for your indices create by snapshot policies that you may have inside Elasticsearch, this is not the same as having machine snapshots of your instances.
@leandrojmp
Yes, I tried upgrading from '7.10.2' to '7.17' before going to '8.4.3'.
Let it be this issue.
I'll try to fix it another way by restoring the snapshot.
I have a 15-day-old ELK server snapshot backup. Indicies are kept in a secondary mount.
I am new to the ELK stack. I'm not interested in old indices and logs. ELK server should now be operational.
Could you please assist high-level steps to restore back my ELK server.
This is not the issue, the issue is that you upgraded to 8.4.3 without fixing the critical issues identified by the upgrade assistant, your cluster is now in version 8.4.3.
root@elk:/usr/share/elasticsearch# sudo ./bin/elasticsearch --version
Version: 8.4.3, Build: deb/42f05b9372a9a4a470db3b52817899b99a76ee73/2022-10-04T07:17:24.662462378Z, JVM: 18.0.2.1
But you use elastalert, and elastalert indices have a mapping that breaks Elasticsearch 8.X, and this is not fixed yet, it will be fixed probably on version 8.5.
So, unfortunately you currently cannot start elasticsearch in version 8.4.3.
This probably won't work, Elasticsearch does not work this way, all the important data is in the data directory, if this is in a secondary mount, you need snapshots of this mount as well.
If you are not interested in old indices, then your best option is to create a new cluster in last version, there are plenty documentation about it.
You cannot restore your cluster now, you need a fix that was not released yet.
@leandrojmp need help to fix above issue.it is still pending
I bring up the services using old snapshot of server. i'm able to open kibana url. unable to find logs in kibana dashboard. Could you please help
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.