Elasticsearch (sort of) losing pipelines on restart

[meerlicht (Daniel)]

hi.... here's my setup:

• ES Cluster of three nodes: one is the ingest node, the two others data nodes.
• It has been setup with 'ansible-elasticsearch'
• All works fine... filebeat is sending data to the ingest node where a pipeline processes the data that is then saved in the data nodes

Once I restart the ingest node the es-logfile logs this (a lot):
java.lang.IllegalArgumentException: pipeline with id [httpd-access] does not exist

Yet,

curl -XGET 'localhost:9200/_ingest/pipeline'

shows it exists with the correct pipeline steps.
My solution currently is to delete and recreate it:

curl -XDELETE 'localhost:9200/_ingest/pipeline/httpd-access'
curl -XPUT 'localhost:9200/_ingest/pipeline/httpd-access' -H 'Content-Type: application/json' -d'...'

As soon as I've recreated it, data is processed again and the errors in the log stop appearing.

I was trying to understand where the pipelines are stored/persisted and investigate there, but so far to no avail. And before attempting to solve it by re-building the cluster I am hoping to get some answers here. Any help is greatly appreciated.

[talevy (Tal Levy)]

that is unexpected,

the Ingest Pipelines are stored in the cluster-state. For whatever reason,
the cluster-state including these pipelines is not being registered in the ingest node's pipeline store.

I will try and reproduce

[Christian_Dahlqvist]

Are all your nodes master eligible? What do you have minimum_master_nodes set to?

[meerlicht (Daniel)]

thanks for you reply.
yes, all are master eligible

curl -XGET 'localhost:9200/_cat/nodes?v&pretty'
ip          heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.1.22           64          81   0    0.01    0.02     0.05 md        -      node02
172.31.1.21           24          80   1    0.00    0.01     0.05 md        *      node01
172.31.1.20           13          99   0    0.00    0.01     0.05 mi        -      node00

minimum_master_nodes ist set to 2 on all three nodes

discovery.zen.minimum_master_nodes: 2

[Christian_Dahlqvist]

That is great. I was wondering of it could be the effect of some kind of split-brain scenario, but then that does not seem to be the case.

[meerlicht (Daniel)]

I'm still trying to understand and made some progress:

• Cluster is setup from scratch; all is fine
• If adding all my pipelines, then restarting elasticsearch I'm back to the error:

java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_2] does not exist

but yet it's there:

curl -XGET 'localhost:9200/_ingest/pipeline/xpack_monitoring_2?pretty'
{
"xpack_monitoring_2" : {
"description" : "2: This is a placeholder pipeline for Monitoring API version 2 so that future versions may fix breaking changes.",
"processors" :
}
}

So: I've started to add my pipelines one by one and found that the error starts as soon as I'm adding a pipeline with a painless-script. Adding the pipeline with the painless-script renders all my pipelines unusable after a restart of the Elasticsearch server ingest node until I delete and re-add all pipelines to the running Elasticsearch server.

I'll try to simplify my setup to then post a sample script & pipeline here.

Additionally this might be related to these topics:

[Ingest Node] pipeline with id [x] does not exist

I think this is the same problem noted here: https://discuss.elastic.co/t/ingest-node-pipeline-with-id-x-does-not-exist-solved/54436 That thread says "solved" but I see no resolution. Just an inability to reproduce the error. We're still getting the error in later versions. Details: ES 5.1.1 Using 2 custom processors Single node cluster(s) Has occurred in at least half a dozen clusters I don't think I could do a better job at noting the specifics than @PatrickKik so I'll quote him:

[Ingest Node] pipeline with id [x] does not exist (solved)

SOLVED Just minutes before the release of 5.0.0-alpha4 I found this bug. I tried it in `5.0.0-alpha4' and it works fine. That's nice. But it still feels like I have lost some time... I'm using elasticsearch-5.0.0-alpha3 and kibana-5.0.0-alpha3. Using Console I register a pipeline: PUT _ingest/pipeline/x { "description": "Ingesting Meetup events", "processors": [ ... ] } When I check if my pipeline exists: GET _ingest/pipeline/x It comes back the way it should: { "pipeline…

[mkrauklis (Michael Krauklis)]

We're still encountering this problem. At least once a week someone on our team runs into this issue and usually re-creating the pipeline resolves it. Occasionally an environment will be more stubborn and will require some combination of node restarts and deletion/recreation of the pipeline to sort things out. Again, we're on 5.1.1 and would love if someone came up with a solution for this (or at least identified/recognized the problem).

[Zachary_Buckholz]

This hit me yesterday when I added new nodes to the cluster and decided to just trust a yum update. When I got mixed version I decided to update the other nodes as well. Now none of my pipelines are working, I have bee troublshooting from the filebeat side thinking it was there. But obviously it's not.