Recommended logic for handling different log files on same host?

#1

I have a host with three different log files. Using one instance of filebeat to ship all of the logs. Started this process back on 6.2 and discovered the "type" option. So based on the log file path I would set the type to say "access, audit, etc..." and ship out to Logstash. The logstash output would then use this type field as part of the index name so they would be easily identifiable in Kibana.

Used this simple code: index => "%{[type]}-%{+YYYY.MM.dd}"

This worked great, until I started adding other types of beats. Take Packetbeat. It comes with this fantastic dashboard for DNS. Well the packetbeat yml files uses types to differentiate the different protocols so now I have indecies that say dns-2019.08.13 and icmp-2019.08.13. All correct since that is how I coded my logstash output, however, the dashboard is expecting packetbeat-2019.08.13 as the index name.

So bottom line, is there another option (metadata?) that I can use to create different custom index names from the same host besides using type?

#2

Use a conditional to build the index name in a metadata field

if <event from packetbeat> {
    mutate { add_field => { "[@metadata][indexName]" => "packetbeat-%{YYYY.MM.dd}" } }
} else {
    mutate { add_field => { "[@metadata][indexName]" => "%{[type]}" } }
}

then in the output use

index => "%{[@metadata][indexName]}

The test for '<event from packetbeat>' would be something like [beat][name] == "packetbeat" but I do not have packetbeat running so I cannot test exactly what it should be.

Do the packetbeat index names not include the version number?

#3

This definitely looks promising. I am pretty ignorant on what @metadata fields are available and if they are consistent across all types of "beats". In regards to your version question, I am not sure if you are referring to the version of the beats but that is available but don't understand how you would use it.

Going to dig into this more.

#4

If I recall correctly, some beats include the version number as well as the beat name in the index name that they expect to use for the dashboards. Thus the dashboards are versioned. You'll know it if it is happening, if not then do not worry about it.

#5

I am searching like crazy, but not having much success finding what default/standard metadata fields exist for beats. Do you know if there is a way to query this information from a running beat? Not to keep adding work so if you don't recall I will keep searching.

#6

This seems to reference some of the fields, but still no definitive list. This looks like it would work with your logic.

https://www.elastic.co/guide/en/logstash/7.2/plugins-inputs-beats.html#plugins-inputs-beats-versioned-indexes

#7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.