I have a host with three different log files. Using one instance of filebeat to ship all of the logs. Started this process back on 6.2 and discovered the "type" option. So based on the log file path I would set the type to say "access, audit, etc..." and ship out to Logstash. The logstash output would then use this type field as part of the index name so they would be easily identifiable in Kibana.
Used this simple code: index => "%{[type]}-%{+YYYY.MM.dd}"
This worked great, until I started adding other types of beats. Take Packetbeat. It comes with this fantastic dashboard for DNS. Well the packetbeat yml files uses types to differentiate the different protocols so now I have indecies that say dns-2019.08.13 and icmp-2019.08.13. All correct since that is how I coded my logstash output, however, the dashboard is expecting packetbeat-2019.08.13 as the index name.
So bottom line, is there another option (metadata?) that I can use to create different custom index names from the same host besides using type?
The test for '<event from packetbeat>' would be something like [beat][name] == "packetbeat" but I do not have packetbeat running so I cannot test exactly what it should be.
Do the packetbeat index names not include the version number?
This definitely looks promising. I am pretty ignorant on what @metadata fields are available and if they are consistent across all types of "beats". In regards to your version question, I am not sure if you are referring to the version of the beats but that is available but don't understand how you would use it.
If I recall correctly, some beats include the version number as well as the beat name in the index name that they expect to use for the dashboards. Thus the dashboards are versioned. You'll know it if it is happening, if not then do not worry about it.
I am searching like crazy, but not having much success finding what default/standard metadata fields exist for beats. Do you know if there is a way to query this information from a running beat? Not to keep adding work so if you don't recall I will keep searching.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.