Current status: Elasticsearch (5.4.1) does not start, stays in Red. The log messages show the following error:
[2018-10-12T18:11:50,720][WARN ][o.e.c.s.ClusterService ] [5awoLm1] failed to notify ClusterStateApplier org.elasticsearch.ElasticsearchParseException: Need [file], [id], or [inline] parameter to refer to scripts
See https://gist.github.com/rsmoorthy/061114e063d9a2ad02f06a128cf68240 for more error details.
There is a bit of history before this error happened.
Step 1: Everything working fine with ELK 541 (ElasticSearch, Logstash, Kibana with the same version), running as a docker container.
Step 2: Upgraded to ELK 640 (running as a docker container again, all components with 6.4.0). ES started fine, then I upgraded to Kibana6 index. Was able to search old indices with new ES and Kibana.
Logstash starts, but wont receive messages and loops the CPU fully. So far troubleshooting failed. (Will raise a separate thread for this, if need be).
Step 3: Hard pressed in time, decided to go back to ELK 541. ELK 541 does not start with the error message as specified at the start.
I am not sure if the Kibana6 index changes is causing this problem. I am okay to remove the kibana6 index (.kibana named) but don't know how - since ES stays in red.
Step 4: Tried to start ELK 640 once more again, but fails completely with lots of error messages. If need be, I can share those messages.
What I am looking for now:
Get ElasticSearch 5.4.1 up and running. Any upgrade issues can be looked at later point (possibly as a separate thread).
Any help will be greatly appreciated, as the server is down.
I did delete the kibana index and restarted ELK, but that does not help. ES stays in red with the same error still in logs.
curl -XDELETE localhost:9200/.kibana
I wonder what this error means;
Need [file], [id], or [inline] parameter to refer to scripts
at org.elasticsearch.ingest.ConfigurationUtils.newConfigurationException(ConfigurationUtils.java:233) ~[elasticsearch-5.4.1.jar:5.4.1]
Checked ingest pipelines and this is what I found (possibly added by ES 640)
I went ahead and deleted the pipelines xpack_monitoring_2 and xpack_monitoring_6
Restarted elasticsearch. Now the error does not appear any more.
org.elasticsearch.ElasticsearchParseException: Need [file], [id], or [inline] parameter to refer to scripts
But ES still stays in red, no further log messages.
So as per cluster allocation explain (https://gist.github.com/rsmoorthy/061114e063d9a2ad02f06a128cf68240#file-es541_cluster_allocation_explain), it shows this message:
"reason" : "Format version is not supported (resource BufferedChecksumIndexInput(SimpleFSIndexInput(path="/var/lib/elasticsearch/nodes/0/indices/xy3ktob4SN-CJoy05XkxWQ/0/index/segments_30"))): 9 (needs to be between 4 and 6)"
So, is it a case that Elasticsearch cannot be downgraded from 6.4.0 to 5.4.1? (Or it cannot be downgraded to any lower version!?)