# \[Filebeat\] Override Module Ingest Pipeline at Runtime **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 4, 2021, 3:57pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432 "2021-11-04T15:57:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 4, 2021, 3:57pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/1 "2021-11-04T15:57:55Z")
# TL;DR Can I override the ingest pipeline for a filebeat module at runtime from `modules.d/module.yml`? # The Rest of the Story I use the `apache` module with filebeat to capture logs on the Apache servers in our environment. On a few servers that a specific application runs on, I get a large number of entries in the `access_log` that can't be parsed by the standard `filebeat-7.x.x-apache-access-pipeline` ingest pipeline. The majority of log entries can be processed by this pipeline, but a relative handful can't, and generate errors. ## The Solution Since this appears to be application-specific, my solution was to copy the `filebeat-7.x.x-apache-access-pipeline` to `filebeat-my-application-apache-access-pipeline` and add an additional grok pattern to process these log entries. This has been successfully tested using `_simulate`. ## The Question Now my question is: Can I override the default ingest pipeline from the `modules.d/apache.yml` file on the impacted hosts, to force them to use the modified pipeline? It seems to me that it **should** be possible, but that certainly doesn't mean it is. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [November 5, 2021, 10:26am UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/2 "2021-11-05T10:26:21Z")
The pipelines are stored in a different directory but yes u could do it from Filebeat or just modify it in Elasticsearch from Kibana after loading it. I would just do it in Kibana as it's easier. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 5, 2021, 11:35am UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/3 "2021-11-05T11:35:39Z")
> [@legoguy1000](#): > > The pipelines are stored in a different directory but yes u could do it from Filebeat or just modify it in Elasticsearch from Kibana after loading it. I would just do it in Kibana as it's easier. I've already created a new pipeline in Elasticsearch with the modifications I need. What I'm trying to do now is tell Elasticsearch to use that pipeline for just these hosts without going in and modifying the actual module itself on the hosts. I.e., how do I modify the value of `[@metadata][pipeline]` to use `filebeat-my-application-apache-access-pipeline` for just this instance in a persistent, repeatable manner so that I don't have to update the `filebeat-x.x.x-apache-access-pipeline` every time we upgrade the filebeat version we're using. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [November 5, 2021, 12:20pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/4 "2021-11-05T12:20:59Z")
Ok ya in the module config u should be able to add `pipeline: xxxxx` and it should override the default module pipeline. ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 5, 2021, 1:23pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/5 "2021-11-05T13:23:18Z")
I'll try it. Do you know where that's documented? I can find it as an option for `output.elasticsearch`, but not at the module level. ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [November 5, 2021, 2:05pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/6 "2021-11-05T14:05:32Z")
See [Allowing users to override pipeline ID in fileset input config by ycombinator · Pull Request #16561 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/16561). ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 5, 2021, 2:50pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/7 "2021-11-05T14:50:43Z")
> [@legoguy1000](#): > > See [Allowing users to override pipeline ID in fileset input config by ycombinator · Pull Request #16561 · elastic/beats · GitHub](https://github.com/elastic/beats/pull/16561). That's exactly what I was looking for, but couldn't find. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 3, 2021, 4:50pm UTC](https://discuss.elastic.co/t/filebeat-override-module-ingest-pipeline-at-runtime/288432/8 "2021-12-03T16:50:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.