Distress: Elasticsearch does not start

Elastic Stack Elasticsearch
1

Current status: Elasticsearch (5.4.1) does not start, stays in Red. The log messages show the following error:

[2018-10-12T18:11:50,720][WARN ][o.e.c.s.ClusterService   ] [5awoLm1] failed to notify ClusterStateApplier
org.elasticsearch.ElasticsearchParseException: Need [file], [id], or [inline] parameter to refer to scripts

See ElasticSearch issues · GitHub for more error details.

There is a bit of history before this error happened.

Step 1: Everything working fine with ELK 541 (Elasticsearch, Logstash, Kibana with the same version), running as a docker container.

Step 2: Upgraded to ELK 640 (running as a docker container again, all components with 6.4.0). ES started fine, then I upgraded to Kibana6 index. Was able to search old indices with new ES and Kibana.

Logstash starts, but wont receive messages and loops the CPU fully. So far troubleshooting failed. (Will raise a separate thread for this, if need be).

Step 3: Hard pressed in time, decided to go back to ELK 541. ELK 541 does not start with the error message as specified at the start.

I am not sure if the Kibana6 index changes is causing this problem. I am okay to remove the kibana6 index (.kibana named) but don't know how - since ES stays in red.

Step 4: Tried to start ELK 640 once more again, but fails completely with lots of error messages. If need be, I can share those messages.

What I am looking for now:

Get Elasticsearch 5.4.1 up and running. Any upgrade issues can be looked at later point (possibly as a separate thread).

Any help will be greatly appreciated, as the server is down.

Update 1:

I did delete the kibana index and restarted ELK, but that does not help. ES stays in red with the same error still in logs.

curl -XDELETE localhost:9200/.kibana
{"acknowledged":true}

I wonder what this error means;

Need [file], [id], or [inline] parameter to refer to scripts
at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:233) ~[elasticsearch-5.4.1.jar:5.4.1]

Update 2:

Checked ingest pipelines and this is what I found (possibly added by ES 640)

See ElasticSearch issues · GitHub

I went ahead and deleted the pipelines xpack_monitoring_2 and xpack_monitoring_6

Restarted elasticsearch. Now the error does not appear any more.

org.elasticsearch.ElasticsearchParseException: Need [file], [id], or [inline] parameter to refer to scripts

But ES still stays in red, no further log messages.

Update 3:

So as per cluster allocation explain (ElasticSearch issues · GitHub), it shows this message:

"reason" : "Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/elasticsearch/nodes/0/indices/xy3ktob4SN-CJoy05XkxWQ/0/index/segments_30"))): 9 (needs to be between 4 and 6)"

So, is it a case that Elasticsearch cannot be downgraded from 6.4.0 to 5.4.1? (Or it cannot be downgraded to any lower version!?)

3

Hi Moorthy,

Upgrades for Elasticsearch are one-way with regard to data. Data written by the 6.x cluster cannot be read by the 5.x cluster, this is likely the cause of all these error messages.

Do you have a snapshot or backup of the data taken prior to the upgrade that to which you could roll back to downgrade to 5.4.1?

4

Hi Lee,

Thanks for your response. I do understand now that downgrade is not possible. I have since then dropped the downgrade plan.

However, I am unable to start it with ES 6.4.0 also. The errors are detailed in https://gist.github.com/rsmoorthy/132223e7540edc2c73c2827f8ebb732d This seems to be the specific message:

org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:305, file:/var/lib/elasticsearch/nodes/0/_state/global-305.st]
..
Caused by: java.lang.IllegalArgumentException: Index patterns must not be null or empty; got null

What should I do to start the ES 6.4.0 back again?

5

Hello everyone,

I resolved the problem as below. This is probably not recommended, but it worked for me (I have a single node ES)

The path /var/lib/elasticsearch/nodes/0/_state folder contains the global information, templates etc. In my case, a file in this folder was corrupted (or unusable). So I simply deleted the folder nodes/0/_state and copied a _state folder from a fresh installation. The nodes/0/indices folder is untouched.

Elastic search came up fine and it DID recognise all the indices (that was in nodes/0/indices folder). It did not contain any of the template settings, which I gladly updated, using already existing scripts. That's all and ES 6.4.0 came up fine.

Thanks folks!

2 Likes
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.