[Filebeat] Override Module Ingest Pipeline at Runtime

Elastic Stack Beats
1

TL;DR

Can I override the ingest pipeline for a filebeat module at runtime from modules.d/module.yml?

The Rest of the Story

I use the apache module with filebeat to capture logs on the Apache servers in our environment. On a few servers that a specific application runs on, I get a large number of entries in the access_log that can't be parsed by the standard filebeat-7.x.x-apache-access-pipeline ingest pipeline. The majority of log entries can be processed by this pipeline, but a relative handful can't, and generate errors.

The Solution

Since this appears to be application-specific, my solution was to copy the filebeat-7.x.x-apache-access-pipeline to filebeat-my-application-apache-access-pipeline and add an additional grok pattern to process these log entries. This has been successfully tested using _simulate.

The Question

Now my question is: Can I override the default ingest pipeline from the modules.d/apache.yml file on the impacted hosts, to force them to use the modified pipeline? It seems to me that it should be possible, but that certainly doesn't mean it is.

2

The pipelines are stored in a different directory but yes u could do it from Filebeat or just modify it in Elasticsearch from Kibana after loading it. I would just do it in Kibana as it's easier.

3

I've already created a new pipeline in Elasticsearch with the modifications I need. What I'm trying to do now is tell Elasticsearch to use that pipeline for just these hosts without going in and modifying the actual module itself on the hosts. I.e., how do I modify the value of [@metadata][pipeline] to use filebeat-my-application-apache-access-pipeline for just this instance in a persistent, repeatable manner so that I don't have to update the filebeat-x.x.x-apache-access-pipeline every time we upgrade the filebeat version we're using.

4

Ok ya in the module config u should be able to add pipeline: xxxxx and it should override the default module pipeline.

5

I'll try it. Do you know where that's documented? I can find it as an option for output.elasticsearch, but not at the module level.

6

See Allowing users to override pipeline ID in fileset input config by ycombinator · Pull Request #16561 · elastic/beats · GitHub.

1 Like
7

That's exactly what I was looking for, but couldn't find. Thanks!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.