If I have several different log files in a directory, and I'm wanting to forward them to logstash for grok'ing and buffering, and then to downstream Elasticsearch. As the files are coming out of Filebeat, how do I tag them with something so that logstash knows which filter to apply? So if I have: …

You can add custom fields to the events that you can then use to conditional filtering in Logstash. You can define multiple prospectors in the Filebeat configuration. So group the files that need the same processing under the same prospector so that the same custom fields are added.

So to do this, I would need to define multiple prospectors? I was really hoping that I could define the paths, and then be able to use some sort of macro/variable within the fields variable. So something like: paths: /path/to/logs/access.log /path/to/other/logs/errors.log fields: %{filename} …

Would the alternative (the custom fields method you suggested, using multiple prospectors) look something like this? filebeat: prospectors: - paths: - /path/to/logs/access.log fields: {log_type: access} - paths: - /path/to/other…

Digging deeper, it seems like I should be able to say: type => "%{[@metadata][log_type]}" or would it be type =>"%{[log_type]}" Or if I were trying to use the source field, using: type => "%{[@metadata][source]}" Does that look right? Is that how I would access a field within logstash, so tha…

The first part is correct on its own, but you may want to change it based on how you intend to write your Logstash config. The second part won't work. You cannot override the type field once it's set. This is documented here . If you want to change the type then set document_type: access in your Fil…

Thank you so much. That will work! [image] andrewkroh: You cannot override the type field once it's set. I'm guessing this is related to the {dynamic_type} in the template file? Is there no way to dynamically set that type based upon the file input? Would I need to create a separate mappi…

[image] jclose: I'm guessing this is related to the {dynamic_type} in the template file? No, it is because the logstash-beats-input doesn't allow it to be overridden if it's already set. [image] jclose: Is there no way to dynamically set that type based upon the file input? In Filebea…

hi , i was new to elk was trying to figure out , how can i differentiate multiple files /var/log/kakfa/*.log these are the files i have in that path controller.log kafka-authorizer.log kafkaServer-gc.log log-cleaner.log state-change.log controller.log controller.log kafka…

bob-bza, your config worked for me but what if you have multiple paths configured in filebeats and you want to have a separate index for each path? For example what if I have this in my filebeat.yml: filebeat:: filebeat.prospectors: input_type: log /var/log/kafka/.log /var/log/message Then…

Discuss the Elastic Stack

How to tag log files in filebeat for logstash ingestion?

Elastic Stack Beats

andrewkroh (Andrew Kroh) March 18, 2016, 1:22pm 4

If you want to apply conditional filtering based on the filename then use the source field.

Topic		Replies	Views
Beat and Logstash Logstash	14	1090	April 12, 2018
Add multiple types in filebeat for logstash output Logstash	4	1864	March 28, 2018
Filter multiple different file beat logs in logstash Logstash	5	23726	March 30, 2017
Identify filebeat input with tags and input type Logstash	2	3138	November 14, 2017
How does the dynamic typing work in filebeat Beats filebeat	2	1372	July 5, 2017

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

How to tag log files in filebeat for logstash ingestion?

Related topics