# Elasticsearch (sort of) losing pipelines on restart
**URL:** https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688
**Category:** Elasticsearch
**Created:** [June 8, 2017, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688 "2017-06-08T08:48:08Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@meerlicht](https://discuss.elastic.co/u/meerlicht)
#### Post date: [June 8, 2017, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/1 "2017-06-08T08:48:08Z")
hi.... here's my setup:
- ES Cluster of three nodes: one is the ingest node, the two others data nodes.
- It has been setup with 'ansible-elasticsearch'
- All works fine... filebeat is sending data to the ingest node where a pipeline processes the data that is then saved in the data nodes
Once I restart the ingest node the es-logfile logs this (a lot):
`java.lang.IllegalArgumentException: pipeline with id [httpd-access] does not exist`
Yet,
> curl -XGET 'localhost:9200/\_ingest/pipeline'
shows it exists with the correct pipeline steps.
My solution currently is to delete and recreate it:
> curl -XDELETE 'localhost:9200/\_ingest/pipeline/httpd-access'
> curl -XPUT 'localhost:9200/\_ingest/pipeline/httpd-access' -H 'Content-Type: application/json' -d'...'
As soon as I've recreated it, data is processed again and the errors in the log stop appearing.
I was trying to understand where the pipelines are stored/persisted and investigate there, but so far to no avail. And before attempting to solve it by re-building the cluster I am hoping to get some answers here. Any help is greatly appreciated.
---
### Author:  [@talevy](https://discuss.elastic.co/u/talevy)
#### Post date: [June 8, 2017, 7:01pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/2 "2017-06-08T19:01:15Z")
that is unexpected,
the Ingest Pipelines are stored in the cluster-state. For whatever reason,
the cluster-state including these pipelines is not being registered in the ingest node's pipeline store.
I will try and reproduce
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 8, 2017, 7:58pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/3 "2017-06-08T19:58:33Z")
Are all your nodes master eligible? What do you have `minimum_master_nodes` set to?
---
### Author:  [@meerlicht](https://discuss.elastic.co/u/meerlicht)
#### Post date: [June 9, 2017, 6:33am UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/4 "2017-06-09T06:33:49Z")
thanks for you reply.
yes, all are master eligible
```
curl -XGET 'localhost:9200/_cat/nodes?v&pretty'
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.1.22 64 81 0 0.01 0.02 0.05 md - node02
172.31.1.21 24 80 1 0.00 0.01 0.05 md * node01
172.31.1.20 13 99 0 0.00 0.01 0.05 mi - node00
```
minimum\_master\_nodes ist set to 2 on all three nodes
```
discovery.zen.minimum_master_nodes: 2
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 9, 2017, 6:35am UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/5 "2017-06-09T06:35:06Z")
That is great. I was wondering of it could be the effect of some kind of split-brain scenario, but then that does not seem to be the case.
---
### Author:  [@meerlicht](https://discuss.elastic.co/u/meerlicht)
#### Post date: [June 18, 2017, 8:25pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/6 "2017-06-18T20:25:02Z")
I'm still trying to understand and made some progress:
- Cluster is setup from scratch; all is fine
- If adding all my pipelines, then restarting elasticsearch I'm back to the error:
`java.lang.IllegalArgumentException: pipeline with id [xpack_monitoring_2] does not exist`
but yet it's there:
> curl -XGET 'localhost:9200/\_ingest/pipeline/xpack\_monitoring\_2?pretty'
> {
> "xpack\_monitoring\_2" : {
> "description" : "2: This is a placeholder pipeline for Monitoring API version 2 so that future versions may fix breaking changes.",
> "processors" :
> }
> }
So: I've started to add my pipelines one by one and found that the error starts as soon as I'm adding a pipeline with a painless-script. Adding the pipeline with the painless-script renders all my pipelines unusable after a restart of the Elasticsearch server ingest node until I delete and re-add all pipelines to the running Elasticsearch server.
I'll try to simplify my setup to then post a sample script & pipeline here.
Additionally this might be related to these topics:
> [@\[Ingest Node\] pipeline with id \[x\] does not exist](https://discuss.elastic.co/t/ingest-node-pipeline-with-id-x-does-not-exist/84570):
>
> I think this is the same problem noted here: [https://discuss.elastic.co/t/ingest-node-pipeline-with-id-x-does-not-exist-solved/54436](https://discuss.elastic.co/t/ingest-node-pipeline-with-id-x-does-not-exist-solved/54436) That thread says "solved" but I see no resolution. Just an inability to reproduce the error. We're still getting the error in later versions. Details: ES 5.1.1 Using 2 custom processors Single node cluster(s) Has occurred in at least half a dozen clusters I don't think I could do a better job at noting the specifics than @PatrickKik so I'll quote him:
> [@\[Ingest Node\] pipeline with id \[x\] does not exist (solved)](https://discuss.elastic.co/t/ingest-node-pipeline-with-id-x-does-not-exist-solved/54436):
>
> SOLVED Just minutes before the release of 5.0.0-alpha4 I found this bug. I tried it in `5.0.0-alpha4' and it works fine. That's nice. But it still feels like I have lost some time... weary I'm using elasticsearch-5.0.0-alpha3 and kibana-5.0.0-alpha3. Using Console I register a pipeline: PUT \_ingest/pipeline/x { "description": "Ingesting Meetup events", "processors": [...] } When I check if my pipeline exists: GET \_ingest/pipeline/x It comes back the way it should: { "pipeline…
---
### Author:  [@mkrauklis](https://discuss.elastic.co/u/mkrauklis)
#### Post date: [June 20, 2017, 1:47pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/7 "2017-06-20T13:47:43Z")
We're still encountering this problem. At least once a week someone on our team runs into this issue and _usually_ re-creating the pipeline resolves it. Occasionally an environment will be more stubborn and will require some combination of node restarts and deletion/recreation of the pipeline to sort things out. Again, we're on 5.1.1 and would love if someone came up with a solution for this (or at least identified/recognized the problem).
---
### Author:  [@Zachary\_Buckholz](https://discuss.elastic.co/u/Zachary_Buckholz)
#### Post date: [July 7, 2017, 7:31pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/8 "2017-07-07T19:31:02Z")
This hit me yesterday when I added new nodes to the cluster and decided to just trust a yum update. When I got mixed version I decided to update the other nodes as well. Now none of my pipelines are working, I have bee troublshooting from the filebeat side thinking it was there. But obviously it's not.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 4, 2017, 7:31pm UTC](https://discuss.elastic.co/t/elasticsearch-sort-of-losing-pipelines-on-restart/88688/9 "2017-08-04T19:31:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.