How to filter data from Filebeats in Logstash

I am designing an ELK stack with ElasticSearch, Logstash and Kibana installed on a dedicated server and Filebeats installed on all the machine generating log files. Sounds like a standard setup.

My question is how to configure that connection between Filebeats and Logstash to post-process logs in Logstash before sending them to ElasticSearch. Specifically, I have two types of log files, JSON-formatted logs from a NodeJS app and Nginx logs. I need a mechanism to distinguish between these two types of input in Logstash.

The examples I could find simply indicate that Logstash receives its inputs from Filebeats. But to convert Nginx files to JSON before insterting them to ElasticSearch I need to distinguish between Filebeats with Nginx logs and Filebeats with Node.js logs. How do I do that?

Or should I be processing logs (by converting them to JSON) in Filebeats before sending them to Logstash?

Thank you,

Michael

It's going to be better to have Filebeat send the files to different ports on the LS side, then you can use tags/types to differentiate and use conditionals when running filters.

It's going to be better to have Filebeat send the files to different ports on the LS side, then you can use tags/types to differentiate and use conditionals when running filters.

Why not set document_type in the Filebeat configuration and have Filebeat send events with the correct type field from the get-go?

1 Like

Good point :slight_smile:

Thank you for your suggestions. I would like to better understand them.

warkolm proposed to run LogStash on multiple ports and configure FileBeat to send each log type to its own port. This sounds like a complicated solution (running multiple processes instead of one) to a simple , or at least, common, problem.

magnusbaeck's suggestion was to set document type in FileBeat. How do I use this document_type in LogStash to filter based on it? Are there examples demonstrating the use of document_type in LogStash?

Are there examples demonstrating the use of document_type in LogStash?

There's lots of documentation on how to do conditional filtering: https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

(But note that the field on the Logstash side won't be called document_type.)