How to calculate new fields to be added in filebeat

Elastic Stack Beats
1

Hi All,
I am relatively new to Elkstack.
I am in the process of setting up Elkstack for analysis of my haproxy configuration on Ubuntu machines.
I have 6.5.4 versions for Filebeat, Elasticsearch and Kibana.

What I need to be able to do is to add a few of fields based upon parsing of the http.request.raw_request_line field.

I need to do 3 things:
1 - parse out the path "/_cluster/health" and put it in a new field
2 - parse out any query params and look for a certain query param and put that into a separate field
3 - map certain paths to a name that will go into another field.

The only way I have been able to find to be able to do this is to add a script to the index. The field I am dealing with is currently a text field, so it seems there will be a pretty huge performance hit if we go down that path.

Is there a way to do this in filebeat without having to preprocess the logs prior to them hitting filebeat?

Thanks

2

As you found out, filebeat on its own has little additional processing functionality. I'd suggest to send the filebeat output to either a Logstash pipeline, or alternatively use an ingest pipeline in Elasticsearch itself.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.