Configure filebeat to send only new events

Elastic Stack Beats
1

Hi,

We had successfully set up the ELK stack to our production environment. We can also see the logs (logs are unstructured) output on our Kibana Server.

Everything is working fine for us. But the only thing we are concerned about is the messages in the kibana are structured for every single line written to a given log file.

Is there a way to merge( clubbing of lines ) the logs messages by max no of line ( in File beat before sending them to logstash or elastic search) which will be consider as 1 event in ElasticSearch / Kibana / Logstash.

Note: Just a note the log messages are unstructured (no particular regex pattern in them). So can't use (https://github.com/elastic/beats/issues/461). But I did tried the max-lines approach, yet the event in kibana shows a document for a single lines.

So is there a way to achieve this?

Thanks

2

If you send the data through Logstash you can use the date filter, followed by the age and drop filters to drop older data. As far as I know Filebeat nor ingest pipelines currently support this.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.