Conditional Filebeat output based on document type

Elastic Stack Beats
1

Currently I have filebeat on a server with multiple different application logs, so I have multiple prospectors grabbing and tagging the different application logs. The logs have different formats so one of them needs to be sent to logstash for filtering will the others are json so they can go directly to Elasticsearch.

My question is, is there a way to ensure the one log that needs filtering is only sent to logstash, not elasticsearch, and the logs that do not need filtering are sent only to Elasticsearch and not logstash. Currently I have something like this for my elasticsearch output

  • index: "audience_backend-%{+yyyy.MM.dd}"
    when.contains:
    type: "audience_backend"
  • index: "audience_exceptions-%{+yyyy.MM.dd}"
    when.contains:
    type: "audience_exceptions"

The issue being is the third log that has a different type is still sent to Elasticsearch under the default filebeat index even though it does not match either of the above.

2

You need to filter out the third log which has a different type.
You could use a processor to drop those unwanted events. (Assuming it is what you want to do with messages not matching anything.)
More on processors: https://www.elastic.co/guide/en/beats/filebeat/master/filtering-and-enhancing-data.html

Let me know if you need further help.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.