# Filebeat is overwriting the pipeline specified in Elastic on start

**URL:** <https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825>\
**Category:** Beats\
**Tags:** filebeat, ingest-pipeline\
**Created:** [March 10, 2021, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825 "2021-03-10T14:58:25Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![JamblaInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamblainc/32/85329_2.png) [@JamblaInc](https://discuss.elastic.co/u/JamblaInc)\
**Post date:** [March 10, 2021, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/1 "2021-03-10T14:58:25Z")

</div>

I am using Filebeat to collect CloudWatch logs and I have modified the ingest node pipeline to extract and index some more information from the logs. However, when Filebeat has restarted the extra processors that I added disappear and it seems the whole pipeline is overwritten. Is there a way to ensure the pipeline isn't altered when starting Filebeat?

I have also observed that when I specify a pipeline in filebeat.yml, Filebeat seems to ignore this and use the default. I define the pipeline as shown below.

I am using Filebeat to collect CloudWatch logs and I have modified the ingest node pipeline to extract and index some more information from the logs. However, when Filebeat has restarted the extra processors that I added disappear and it seems the whole pipeline is overwritten. Is there a way to ensure the pipeline isn't altered when starting Filebeat?

I have also observed that when I specify a pipeline in filebeat.yml, Filebeat seems to ignore this and use the default. I define the pipeline as shown below.

```auto
output.elasticsearch:
    hosts: ["127.0.0.1:9243"]
    pipeline: "filebeat-7.11.0-aws-cloudtrail-pipeline-test"

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 11, 2021, 12:47am UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/2 "2021-03-11T00:47:07Z")

</div>

Hi @JamblaInc Welcome to the community.

There is a little subtle magic to this I think... the default pipeline is used and overrides what you are specifying in the elasticsearch output, I believe you will need to define it in the input sections

So first are you using the the [AWS Module](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-aws.html) and / or the [AWS Cloudwatch Input](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-cloudwatch.html) or both?

If you are using the AWS CloudWatch input you would specify it there

See [pipeline](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-cloudwatch.html#_pipeline) ... and I think the magic there is actually a default value for it and then if you read the little section below it says

_The pipeline ID can also be configured in the Elasticsearch output, but this option usually results in simpler configuration files. If the pipeline is configured both in the input and output, the option from the input is used._

Which means the default pipeline will always override the output section.

This may not be it but take a look let me us know...

There was a [similar discuss](https://discuss.elastic.co/t/configuring-pipeline-in-filebeat-module-nginx/162561) a while back...

---

<div class="post-metadata">

**Author:** ![JamblaInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamblainc/32/85329_2.png) [@JamblaInc](https://discuss.elastic.co/u/JamblaInc)\
**Post date:** [March 11, 2021, 10:57am UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/3 "2021-03-11T10:57:19Z")

</div>

Thanks for the reply, I am using the AWS Module but I don't see an option to define a pipeline in there. So where exactly do I define it?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 11, 2021, 2:35pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/4 "2021-03-11T14:35:20Z")

</div>

Can you share your sanitized config? It looks like it is missing in the docs or we might need to do something else. It is still beta

I think under the same level as enabled, like this

```
- module: aws
  cloudtrail:
    enabled: false
    input:
      pipeline: my-pipeline
```

---

<div class="post-metadata">

**Author:** ![JamblaInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamblainc/32/85329_2.png) [@JamblaInc](https://discuss.elastic.co/u/JamblaInc)\
**Post date:** [March 12, 2021, 8:35am UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/5 "2021-03-12T08:35:37Z")

</div>

This didn't work either, it's still loading the default pipeline.

My aws.yml:

```auto
- module: aws
  cloudtrail:
    enabled: true
    var.queue_url: URL
    input:
      pipeline: filebeat-7.11.0-aws-cloudtrail-pipeline-test

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 12, 2021, 3:40pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/6 "2021-03-12T15:40:39Z")

</div>

Apologies I don't have a direct answer but... Hmmm.... you up for a little debugging unfortunately I don't have an aws test bed handy

So lets take a look at with a little more verbose logging.

You can run filebeat in the foreground with these parameters, warning this will be quite verbose I would only have like cloudtrail enable so we can cut it down ... let it run untill you see it processing messages then kill it.

Run filebeat in the foreground When you run this it will dump a lot of config... if you have

`filebeat -e -d "*"` or `./filebeat -e -d "*"` depending on how you installed you

It will tell you it is loaded the default piplines with log lines like this.. don't let that distract you, it will always say that.

```
2021-03-12T07:32:17.519-0800 DEBUG [esclientleg] eslegclient/connection.go:364 GET http://localhost:9200/_nodes/ingest <nil>
2021-03-12T07:32:17.524-0800 DEBUG [esclientleg] eslegclient/connection.go:364 GET http://localhost:9200/_ingest/pipeline/filebeat-7.11.1-nginx-access-pipeline <nil>
2021-03-12T07:32:17.527-0800 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.11.1-nginx-access-pipeline already loaded
2021-03-12T07:32:17.527-0800 DEBUG [modules] fileset/pipelines.go:67 Required processors: []
2021-03-12T07:32:17.527-0800 DEBUG [esclientleg] eslegclient/connection.go:364 GET http://localhost:9200/_ingest/pipeline/filebeat-7.11.1-nginx-error-pipeline <nil>
2021-03-12T07:32:17.531-0800 DEBUG [modules] fileset/pipelines.go:120 Pipeline filebeat-7.11.1-nginx-error-pipeline already loaded

```

But what we want to see is a couple of the messages...

The header of each should look like this, we want to see what that pipeline value is.

```
2021-03-12T07:35:55.708-0800 DEBUG [processors] processing/processors.go:203 Publish event: {
  "@timestamp": "2021-03-12T15:35:55.708Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.11.1",
    "pipeline": "my-pipeline" <------- This Value 
  },
  "log": {
    "offset": 790,
    "file": {
      "path": "/Users/sbrown/workspace/sample-data/nginx/nginx-5rows.log"
    }
  },

```

Let me know what you see.

---

<div class="post-metadata">

**Author:** ![JamblaInc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jamblainc/32/85329_2.png) [@JamblaInc](https://discuss.elastic.co/u/JamblaInc)\
**Post date:** [March 15, 2021, 1:07pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/7 "2021-03-15T13:07:51Z")

</div>

So this is showing the correct pipeline. I had a look at the events and it seems they are now being properly processed. I think the previous step may have fixed it.

Thank you for resolving this, it seems the only outstanding issue is if the default pipeline is used it is overwritten every time filebeat is restarted.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 15, 2021, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/8 "2021-03-15T14:54:19Z")

</div>

Good to hear,

Yes if you edit the default pipeline that will happen... I do think there is a way to stop that as well with setting managing the template to false (or some other setting I would need to check, and that might cause other unintended consequences) , but editing the default pipeline is probably not the best practice as there is a lot of logic to get modules back to a working state / default state. After all, by definition it is the default pipeline is just that, what you created is a custom pipeline. 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2021, 6:50am UTC](https://discuss.elastic.co/t/filebeat-is-overwriting-the-pipeline-specified-in-elastic-on-start/266825/10 "2021-04-13T06:50:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.