How to restart elasticsearch after a power outage?

Elastic Stack Elasticsearch
1

I have been hosting elasticsearch locally on my computer for over 1 year. It has been operating perfectly. Earlier this morning, there was a power outage in my city which caused my computer to lose power and unexpectedly shutdown. When I got electricity back, I started up my computer. I ran the command systemctl start elasticsearch, but it gave this error:

Jan 03 16:54:31 elk systemd-entrypoint[586]: Jan 03, 2025 4:54:31 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Jan 03 16:54:31 elk systemd-entrypoint[586]: WARNING: COMPAT locale provider will be removed in a future release
Jan 03 16:54:47 elk systemd-entrypoint[586]: ERROR: Elasticsearch did not exit normally - check the logs at /var/log/elasticsearch/demo.log
Jan 03 16:54:48 elk systemd-entrypoint[586]: ERROR: Elasticsearch died while starting up, with exit code 1
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Jan 03 16:54:48 elk systemd[1]: Failed to start Elasticsearch.
Jan 03 16:54:48 elk systemd[1]: elasticsearch.service: Consumed 42.521s CPU time.

When I did a cat /var/log/elasticsearch/demo.log, I saw this errors:


[2025-01-03T16:54:47,623][ERROR][o.e.b.Elasticsearch      ] [elk] fatal exception while booting Elasticsearch
java.lang.ExceptionInInitializerError: null
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer.<init>(BulgarianAnalyzer.java:88) ~[lucene-analysis-common-9.11.1.jar:?]
        at org.elasticsearch.index.analysis.PreBuiltAnalyzerProviderFactory.<init>(PreBuiltAnalyzerProviderFactory.java:45) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.analysis.common.CommonAnalysisPlugin.getPreBuiltAnalyzerProviderFactories(CommonAnalysisPlugin.java:419) ~[?:?]
        at org.elasticsearch.indices.analysis.AnalysisModule.setupPreBuiltAnalyzerProviderFactories(AnalysisModule.java:175) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.indices.analysis.AnalysisModule.<init>(AnalysisModule.java:88) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.NodeConstruction.createAnalysisRegistry(NodeConstruction.java:623) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:275) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:192) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:242) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:242) ~[elasticsearch-8.15.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:76) ~[elasticsearch-8.15.0.jar:?]
Caused by: java.io.UncheckedIOException: Unable to load default stopword set
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer$DefaultSetHolder.<clinit>(BulgarianAnalyzer.java:79) ~[lucene-analysis-common-9.11.1.jar:?]
        ... 11 more
Caused by: java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:279) ~[?:?]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:405) ~[?:?]
        at sun.nio.cs.StreamDecoder.lockedRead(StreamDecoder.java:217) ~[?:?]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:171) ~[?:?]
        at java.io.InputStreamReader.read(InputStreamReader.java:186) ~[?:?]
        at java.io.BufferedReader.fill(BufferedReader.java:160) ~[?:?]
        at java.io.BufferedReader.implReadLine(BufferedReader.java:370) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:347) ~[?:?]
        at java.io.BufferedReader.readLine(BufferedReader.java:436) ~[?:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:121) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:145) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:175) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.WordlistLoader.getWordSet(WordlistLoader.java:159) ~[lucene-core-9.11.1.jar:?]
        at org.apache.lucene.analysis.bg.BulgarianAnalyzer$DefaultSetHolder.<clinit>(BulgarianAnalyzer.java:71) ~[lucene-analysis-common-9.11.1.jar:?]
        ... 11 more

How do I get elasticsearch to work again?

Notes: I am using version 8 of elasticsearch. It is only a single node.

2

Wild guess, but when the system lost power you also lost some data, effectively corrupting some files. Do you recall if you were asked to do a filesystem check when it came back? And if it "fixed" any errors.

Aside from the ERROR log shown, were there any others? Or even some warnings you maybe thought were not significant.

"Unable to load default stopword set"
Caused by: java.nio.charset.MalformedInputException: Input length = 1

looks like it could not read a file, and that file (and maybe others) is corrupted. Where specifically the BulgarianAnalyzer would be looking I dont know.

You might want to extract the same elasticsearch release you had into a side/parallel directory, and do a recursive binary diff on both trees to see if any files differ. config files will, certs, etc, but that should be it.

You would need to be lucky for it to just one file. And thats just the elasticsearch tree, your indices could also be damaged.

Or it might be something else entirely of course.

For me, I get this from the recursive diff.

diff -r -q ~/ELK/elasticsearch-8.17.0 ~/ELK/side/elasticsearch-8.17.0

Only in /Users/vf/ELK/elasticsearch-8.17.0/config: certs

Only in /Users/vf/ELK/elasticsearch-8.17.0/config: elasticsearch.keystore

Files /Users/vf/ELK/elasticsearch-8.17.0/config/elasticsearch.yml and /Users/vf/ELK/side/elasticsearch-8.17.0/config/elasticsearch.yml differ

Files /Users/vf/ELK/elasticsearch-8.17.0/config/jvm.options and /Users/vf/ELK/side/elasticsearch-8.17.0/config/jvm.options differ

which is what I would expect.