Elasticsearch service getting failed on Ubuntu 20.4

Elastic Stack Elasticsearch
1

Hi, I have installed elasticsearch 7.1.0 on Ubuntu 20.4 with java-11-openjdk-amd64. Since I am not a developer hence I expect respect for this post.

Elasticsearch service is getting started successfully whenever I run;

sudo -i service elasticsearch start

and after 3 seconds when I check the status here is the output;

</>
root@server:~# sudo -i service elasticsearch status
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2022-10-28 16:24:31 PKT; 28s ago
Docs: http://www.elastic.co
Process: 6851 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 6851 (code=exited, status=1/FAILURE)

Oct 28 16:24:28 server.rni.com.pk systemd[1]: Started Elasticsearch.
Oct 28 16:24:31 server.rni.com.pk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Oct 28 16:24:31 server.rni.com.pk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
</>

I checked for Elastic Search Logs and found below details but unable to find any help with regards to the same;

</>

[2022-10-28T15:53:55,440][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [server.xyz.com.pk] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read /var/lib/elasticsearch/nodes/0/_state/node-14.st]; nested: IOException[failed to read /var/lib/elasticsearch/nodes/0/_state/node-14.st]; nested: XContentParseException[[-1:36] [node_meta_data] unknown field [node_version], parser not found];
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.1.0.jar:7.1.0]
Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read /var/lib/elasticsearch/nodes/0/_state/node-14.st
at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:414) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:406) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
... 6 more
Caused by: java.io.IOException: failed to read /var/lib/elasticsearch/nodes/0/_state/node-14.st
at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:408) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:406) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
... 6 more
Caused by: org.elasticsearch.common.xcontent.XContentParseException: [-1:36] [node_meta_data] unknown field [node_version], parser not found
at org.elasticsearch.common.xcontent.ObjectParser.getParser(ObjectParser.java:369) ~[elasticsearch-x-content-7.1.0.jar:7.1.0]
at org.elasticsearch.common.xcontent.ObjectParser.parse(ObjectParser.java:158) ~[elasticsearch-x-content-7.1.0.jar:7.1.0]
at org.elasticsearch.common.xcontent.ObjectParser.apply(ObjectParser.java:182) ~[elasticsearch-x-content-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:110) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeMetaData$1.fromXContent(NodeMetaData.java:94) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:302) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadGeneration(MetaDataStateFormat.java:404) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:433) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:454) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.loadOrCreateNodeMetaData(NodeEnvironment.java:406) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:302) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:272) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.node.Node.(Node.java:252) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:211) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:325) ~[elasticsearch-7.1.0.jar:7.1.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.1.0.jar:7.1.0]
... 6 more
</>

I checked my other Centos Servers and found that Elasticsearch is running smoothly for more than 2 months but I am unable to have this consistency on this machine. Any precious advice for this will be very helpful.

Thanks.

2

elasticsearch 7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

when I am trying to upgrade Elasticsearch the service is not starting at all.

4

Welcome to our community! :smiley:

Are you trying to install this as a new instance?

5

yes, we have already installed this as a new instance.

6

Is there a reason you are using an EOL version?

7

well its the developer's choice.

8

You should remind them that Elasticsearch 7.1 was released in early 2019 and has been EOL for quite a while. If you or they run into continuing issues, you are likely going to struggle to get anyone to assist you without upgrading to a more current version.

However I would start by removing Elasticsearch, deleting /var/lib/elasticsearch and then try resintalling.

10

We have already deleted and reinstalled Elasticsearch number of times

11

Then something else is happening if it's constantly causing these IO errors, have you checked the hardware?

12

Its actually a cloud virtual machine and we have reinstalled the OS 3-4 times already.

13

What sort of storage is it?

14

SSD Storage

15

If it is a cloud VM it is probably not local SSDs. Can you provide more details?

You also mentioned having some CentOS servers running just fine. Are they using the same hardware?

16

Those CentOS servers are physical servers.

17

But I haven't heard anywhere that CloudVM can't run Elasticsearch

18

You can run Elasticsearch in the cloud. Most people do. Not every cloud instance might however be suitable, which is why I am asking for details.

I have seen users having problems trying to install and run Elasticsearch on cloud VMs with limits or restrictions in place, which may not always be obvious.

19

any process to have that checked out ?

20

How is the storage mounted? As it is a cloud VM I assume it is networked storage. What type of networked storage is used?

No, not that I am aware of. It generally just work for most major cloud providers.

21

No idea about that because we are only using a number of VM instances from them and none of them are having any issues with any services but here in this particular scenario we are getting stuck.