How to restart elasticsearch after a power outage?

Elastic Stack Elasticsearch
1

I have been hosting elasticsearch locally on my computer for over 1 year. It has been operating perfectly. Earlier this morning, there was a power outage in my city which caused my computer to lose power and unexpectedly shutdown. When I got electricity back, I started up my computer. I ran the command systemctl start elasticsearch, but it gave this error:

Jan 03 16:54:31 elk systemd-entrypoint[586]: Jan 03, 2025 4:54:31 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Jan 03 16:54:31 elk systemd-entrypoint[586]: WARNING: COMPAT locale provider will be removed in a future release
Jan 03 16:54:47 elk systemd-entrypoint[586]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/demo.log
Jan 03 16:54:48 elk systemd-entrypoint[586]: ERROR: Elasticsearch died while starting up, with exit code 1
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jan 03 16:54:48 elk systemd[1]: Failed to start Elasticsearch.
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Consumed 42.521s CPU time.

When I did a cat /var/log/elasticsearch/demo.log, I saw this errors:


[2025-01-03T16:54:47,623][ERROR][o.e.b.Elasticsearch      ] [elk] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer.<init>(BulgarianAnalyzer.java:88) ~[lucene-analysis-common-9.11.1.jar:?]
        at org.elasticsearch.index.analysis.PreBuiltAnalyzerProviderFactory.<init>(PreBuiltAnalyzerProviderFactory.java:45) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.analysis.common.CommonAnalysisPlugin.getPreBuiltAnalyzerProviderFactories(CommonAnalysisPlugin.java:419) ~[?:?]
        at org.elasticsearch.indices.analysis.AnalysisModule.setupPreBuiltAnalyzerProviderFactories(AnalysisModule.java:175) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.indices.analysis.AnalysisModule.<init>(AnalysisModule.java:88) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.NodeConstruction.createAnalysisRegistry(NodeConstruction.java:623) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:275) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:192) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:242) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:242) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:76) ~[elasticsearch-8.15.0.jar:?]
Caused by: java.io.UncheckedIOException: Unable to load default stopword set
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer$DefaultSetHolder.<clinit>(BulgarianAnalyzer.java:79) ~[lucene-analysis-common-9.11.1.jar:?]
        ... 11 more
Caused by: java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:279) ~[?:?]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:405) ~[?:?]
        at sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217) ~[?:?]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171) ~[?:?]
        at java.io.InputStreamReader.read(InputStreamReader.java:186) ~[?:?]
        at java.io.BufferedReader.fill(BufferedReader.java:160) ~[?:?]
        at java.io.BufferedReader.implReadLine(BufferedReader.java:370) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:347) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:436) ~[?:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:121) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:145) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:175) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:159) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer$DefaultSetHolder.<clinit>(BulgarianAnalyzer.java:71) ~[lucene-analysis-common-9.11.1.jar:?]
        ... 11 more

How do I get elasticsearch to work again?

Notes: I am using version 8 of elasticsearch. It is only a single node.

2

Wild guess, but when the system lost power you also lost some data, effectively corrupting some files. Do you recall if you were asked to do a filesystem check when it came back? And if it "fixed" any errors.

Aside from the ERROR log shown, were there any others? Or even some warnings you maybe thought were not significant.

"Unable to load default stopword set"
Caused by: java.nio.charset.MalformedInputException: Input length = 1

looks like it could not read a file, and that file (and maybe others) is corrupted. Where specifically the BulgarianAnalyzer would be looking I dont know.

You might want to extract the same elasticsearch release you had into a side/parallel directory, and do a recursive binary diff on both trees to see if any files differ. config files will, certs, etc, but that should be it.

You would need to be lucky for it to just one file. And thats just the elasticsearch tree, your indices could also be damaged.

Or it might be something else entirely of course.

For me, I get this from the recursive diff.

diff -r -q ~/ELK/elasticsearch-8.17.0 ~/ELK/side/elasticsearch-8.17.0

Only in /Users/vf/ELK/elasticsearch-8.17.0/config: certs

Only in /Users/vf/ELK/elasticsearch-8.17.0/config: elasticsearch.keystore

Files /Users/vf/ELK/elasticsearch-8.17.0/config/elasticsearch.yml and /Users/vf/ELK/side/elasticsearch-8.17.0/config/elasticsearch.yml differ

Files /Users/vf/ELK/elasticsearch-8.17.0/config/jvm.options and /Users/vf/ELK/side/elasticsearch-8.17.0/config/jvm.options differ

which is what I would expect.

3

@learningelastic - did you recover your system?

4

Thank you for your suggestion! You successfully guessed some of my earlier challenges related to corrupted files!

My elasticsearch is hosted as a proxmox container. During the proxmox boot up, I was getting some corrupted files or corrupted block errors. I ran some fsck commands suggested by some blog posts. This automatically fixed some issues and allowed my elasticsearch container to finally boot up.

Once the elasticsearch container boot up, that's when I was able to SSH into the container and run the systemctl start elasticsearch comand, only to run into the errors posted in my original message.

So I guess I'l have to brute force repair remaining elasticsearch files as you suggested....ugh....

UPDATE --- oh wait, for some reason, elasticsearch is up and running again! I didn't do anything differently yet. But I can finally do a curl -k -u username:pass https://localhost:9200/ and query things. Just Kibana is down for some reason....this is promising....

5

After a few systemctl restart kibana, kibana is back up and running as well! So it looks like everything magically repaired itself after several reboots! Thanks for your insight!

6

I still suggest to diff your tree with the distribution files.