How to filter data from Filebeats in Logstash

Elastic Stack Logstash
1

I am designing an ELK stack with ElasticSearch, Logstash and Kibana installed on a dedicated server and Filebeats installed on all the machine generating log files. Sounds like a standard setup.

My question is how to configure that connection between Filebeats and Logstash to post-process logs in Logstash before sending them to ElasticSearch. Specifically, I have two types of log files, JSON-formatted logs from a NodeJS app and Nginx logs. I need a mechanism to distinguish between these two types of input in Logstash.

The examples I could find simply indicate that Logstash receives its inputs from Filebeats. But to convert Nginx files to JSON before insterting them to ElasticSearch I need to distinguish between Filebeats with Nginx logs and Filebeats with Node.js logs. How do I do that?

Or should I be processing logs (by converting them to JSON) in Filebeats before sending them to Logstash?

Thank you,

Michael

2

It's going to be better to have Filebeat send the files to different ports on the LS side, then you can use tags/types to differentiate and use conditionals when running filters.

3

It's going to be better to have Filebeat send the files to different ports on the LS side, then you can use tags/types to differentiate and use conditionals when running filters.

Why not set document_type in the Filebeat configuration and have Filebeat send events with the correct type field from the get-go?

1 Like
4

Good point :slight_smile:

5

Thank you for your suggestions. I would like to better understand them.

warkolm proposed to run LogStash on multiple ports and configure FileBeat to send each log type to its own port. This sounds like a complicated solution (running multiple processes instead of one) to a simple , or at least, common, problem.

magnusbaeck's suggestion was to set document type in FileBeat. How do I use this document_type in LogStash to filter based on it? Are there examples demonstrating the use of document_type in LogStash?

6

Are there examples demonstrating the use of document_type in LogStash?

There's lots of documentation on how to do conditional filtering: Accessing event data and fields | Logstash Reference [8.11] | Elastic

(But note that the field on the Logstash side won't be called document_type.)