If I have several different log files in a directory, and I'm wanting to forward them to logstash for grok'ing and buffering, and then to downstream Elasticsearch.
As the files are coming out of Filebeat, how do I tag them with something so that logstash knows which filter to apply?
You can add custom fields to the events that you can then use to conditional filtering in Logstash.
You can define multiple prospectors in the Filebeat configuration. So group the files that need the same processing under the same prospector so that the same custom fields are added.
If so, what are the options on the logstash side for setting the field type in the 'input' for beats, so that I can tag appropriately, so that my filter easily directs each message to the appropriate grok (I'm still learning about the macros available)?
I could see wanting do say something like:
input {
beats {
port => 5044
type => %{log_type}
}
}
Thanks for your help, by the way. I haven't been able to find any explicit examples on how to handle sending messages from beats to logstash, where logstash will handle the grok and parsing into JSON for downstream Elastic.
The first part is correct on its own, but you may want to change it based on how you intend to write your Logstash config.
The second part won't work. You cannot override the type field once it's set. This is documented here. If you want to change the type then set document_type: access in your Filebeat configuration.
Here's an example Logstash config based on the Filebeat config you gave:
I'm guessing this is related to the {dynamic_type} in the template file? Is there no way to dynamically set that type based upon the file input? Would I need to create a separate mapping for each of the different log types, identify those mappings in the yml file, and then do it that way?
In the end, with what you've shown me, I don't think it matters much. As long as I can quickly filter and grok based upon the incoming message type, then I'm fine. I'm just trying to learn a little more about how filebeat works at this point.
No, it is because the logstash-beats-input doesn't allow it to be overridden if it's already set.
In Filebeat you can use document_type: mytype on a per prospector basis. And in Logstash there are many ways to do this. My personal preference would be set type at the source using document_type. But here's an arbitrary Logstash example.
hi , i was new to elk was trying to figure out , how can i differentiate multiple files
/var/log/kakfa/*.log
these are the files i have in that path
controller.log kafka-authorizer.log kafkaServer-gc.log log-cleaner.log state-change.log controller.log controller.log kafka-request.log server.log
Then how do I configure my logstash config file so that the kafka logs will be associated to the "kafka" index and the message logs will be associated to "message" index?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.