Pipeline with id [xpack_monitoring_6] does not exist

Elastic Stack Elasticsearch
1

I have a 5.5.1 ES cluster that I was looking to upgrade to 5.6.16. It is a 4 node cluster with 1 master node and 3 data nodes in TEST (non-prod) environment. In order to test my upgrade scripts, I added a new master node and upgraded the master node alone to 5.6.16. The rest of the nodes i.e. 3 data nodes + 1 master node were NOT yet upgraded. All went well. After a couple of days, it looks like the ingest pipelines stopped working even though GET showed that pipelines are indeed there. I had a logstash that made use of a pipeline while indexing to ES. That stopped working since indexing to ES would fail with pipeline with id [my_pipeline] does not exist. I removed the lone master node that had been upgraded to 5.6.16. I temporarily removed the use of pipeline in logstash as well. Now the logs are full of errors like below:

[2019-11-30T03:45:39,568][DEBUG][o.e.a.b.TransportBulkAction] [abc.reddog.microsoft.com] failed to execute pipeline [xpack_monitoring_6] for document [.monitoring-es-6-2019.11.30/doc/_hOZAPXgRqqGd6c6_feEDg:_aIGyBOqRH6NUrYMjjjU7w:my_index:0:p]
java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_6] does not exist
	at org.elasticsearch.ingest.PipelineExecutionService.getPipeline(PipelineExecutionService.java:194) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.ingest.PipelineExecutionService.access$100(PipelineExecutionService.java:41) ~[elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.ingest.PipelineExecutionService$2.doRun(PipelineExecutionService.java:88) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.1.jar:5.5.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.1.jar:5.5.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]
	at java.lang.Thread.run(Thread.java:748)

Because of this, the monitoring stats and details aren't visible since it's xpack_monitoring_6 pipeline.

This is what I've tried:

  1. Created a new pipeline. Success
  2. GET on the new pipeline. Success
  3. GET on existing pipeline. Success
  4. Simulate new pipeline. FAILURE. SIMULATE API fails with pipeline with id [new_pipeline] does not exist.
  5. Deleted the existing pipeline and recreated it but still SIMULATE API fails with pipeline with id [my_pipeline] does not exist.
  6. Deleted the existing pipeline, did rolling restart of entire cluster and recreated it but still SIMULATE API fails with pipeline with id [my_pipeline] does not exist.
  7. Removed the lone master node that was upgraded to 5.6.16. Deleted pipeline. Did rolling restart. Still SIMULATE API fails with pipeline with id [my_pipeline] does not exist.

Can someone help me understand the root cause of this issue and what can I can to resolve this issue. Because of this issue:

  1. Monitoring details are lost.
  2. I've had to comment pipeline usage in my logstash else indexing wouldn't happen.
  3. I cannot go ahead with upgrading to 5.6.x
2

@Christian_Dahlqvist any thoughts on this?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.