My elasticsearch is not running with error code 128

Hi.
I'm using Elasticsearch 8.6.2 on Ubuntu 22.04.
Since a few days ago, my elastic is not running.
Please help me.

The system output is below.

dev@logserver:~$ sudo systemctl status elasticsearch
× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2023-03-17 09:45:13 CST; 1min 57s ago
       Docs: https://www.elastic.co
    Process: 2133 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=128)
   Main PID: 2133 (code=exited, status=128)
        CPU: 1min 30.382s

Mar 17 09:44:43 logserver systemd[1]: Starting Elasticsearch...
Mar 17 09:45:03 logserver systemd[1]: Started Elasticsearch.
Mar 17 09:45:13 logserver systemd-entrypoint[2133]: ERROR: Elasticsearch exited unexpectedly
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Main process exited, code=exited, status=128/n/a
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Consumed 1min 30.382s CPU time.

And I have elasticsearch.log, but I don't know how can share it.

Please help me.

Hi @Terry_2018 Welcome to the community!

Pastebin... Or a gist

Try running

journalctl -u elasticsearch -f

Mar 17 09:41:59 logserver systemd-entrypoint[1876]: ERROR: Elasticsearch exited unexpectedly
Mar 17 09:41:59 logserver systemd[1]: elasticsearch.service: Main process exited, code=exited, status=128/n/a
Mar 17 09:41:59 logserver systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mar 17 09:41:59 logserver systemd[1]: elasticsearch.service: Consumed 1min 36.959s CPU time.
Mar 17 09:44:43 logserver systemd[1]: Starting Elasticsearch...
Mar 17 09:45:03 logserver systemd[1]: Started Elasticsearch.
Mar 17 09:45:13 logserver systemd-entrypoint[2133]: ERROR: Elasticsearch exited unexpectedly
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Main process exited, code=exited, status=128/n/a
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Mar 17 09:45:13 logserver systemd[1]: elasticsearch.service: Consumed 1min 30.382s CPU time.

You won't see the error with journalctl, you need to look directly in the syslog file, check /var/log/syslog for any hint of why the systemd server cannot start the elasticsearch service.

You may try to start it again to get some fresh logs.

Yup you are right ... other components do elasticsearch not...

Assuming you installed with defaults...

tail -f /var/log/elasticsearch/elasticsearch.log

I have entire elasticsearch.log, how can I share it?

		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1589) ~[?:?]
	Caused by: java.lang.InternalError: a fault occurred in an unsafe memory access operation
		at org.apache.lucene.store.BufferedChecksumIndexInput.readBytes(BufferedChecksumIndexInput.java:46) ~[lucene-core-9.4.2.jar:?]
		at org.apache.lucene.store.DataInput.readBytes(DataInput.java:72) ~[lucene-core-9.4.2.jar:?]
		at org.apache.lucene.store.ChecksumIndexInput.skipByReading(ChecksumIndexInput.java:79) ~[lucene-core-9.4.2.jar:?]
		at org.apache.lucene.store.ChecksumIndexInput.seek(ChecksumIndexInput.java:64) ~[lucene-core-9.4.2.jar:?]
		at org.apache.lucene.store.IndexInput.skipBytes(IndexInput.java:89) ~[lucene-core-9.4.2.jar:?]
		at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:489) ~[lucene-core-9.4.2.jar:?]
		... 22 more

Hi @Terry_2018

I think we need to back up.

Exactly how did you install?

How many nodes?

Is security Enabled?

What is the size of the server? CPU RAM

Are you running other applications / processes?

Did you do anything in the data path? Move files touch them etc?

It looks like there are corrupted indices / data

1 Like

That log has several errors with a cause like this:

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[.geoip_databases][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469) ~[elasticsearch-8.6.2.jar:?]
	... 190 more

or

Caused by: org.elasticsearch.action.search.SearchPhaseExecutionException: Search rejected due to missing shards [[metrics-endpoint.metadata_current_default][0]]. Consider using `allow_partial_search_results` setting to bypass this error.
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.run(AbstractSearchAsyncAction.java:245) ~[elasticsearch-8.6.2.jar:?]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:469) ~[elasticsearch-8.6.2.jar:?]
	... 97 more

I also suspect that files in the node's data path may have been modified or deleted.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.