How to calculate new fields to be added in filebeat

Hi All,
I am relatively new to Elkstack.
I am in the process of setting up Elkstack for analysis of my haproxy configuration on Ubuntu machines.
I have 6.5.4 versions for Filebeat, Elasticsearch and Kibana.

What I need to be able to do is to add a few of fields based upon parsing of the http.request.raw_request_line field.

I need to do 3 things:
1 - parse out the path "/_cluster/health" and put it in a new field
2 - parse out any query params and look for a certain query param and put that into a separate field
3 - map certain paths to a name that will go into another field.

The only way I have been able to find to be able to do this is to add a script to the index. The field I am dealing with is currently a text field, so it seems there will be a pretty huge performance hit if we go down that path.

Is there a way to do this in filebeat without having to preprocess the logs prior to them hitting filebeat?


As you found out, filebeat on its own has little additional processing functionality. I'd suggest to send the filebeat output to either a Logstash pipeline, or alternatively use an ingest pipeline in Elasticsearch itself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.