Output to indexes based on [agent.type] (or any conditionals) not working

Elastic Stack Logstash
1

Sorry to ask this, I have searched long for an answer but not found anything.
My winlogbeat, packetbeat and metricbeat are all sent to logstash on 5044. I want logstash to output them to dedicated indexes. I am using the following config for logstash but everything goes to the last index. Please tell me what have I got wrong, thanks!

input {
beats {
port => "5044"
}
}

filter {
if [agent.type] == "metricbeat" {
mutate { add_field => { "[@metadata][target_index]" => "metricbeat-%{+YYYY.MM.dd}" } }
} else if [agent.type] == "packetbeat" {
mutate { add_field => { "[@metadata][target_index]" => "packetbeat-%{+YYYY.MM.dd}" } }
} else {
mutate { add_field => { "[@metadata][target_index]" => "winlogbeat-%{+YYYY.MM.dd}" } }
}
}

output {
elasticsearch {
hosts => [""]
cacert => "C:\logstash\config\certsESSOC\elasticsearch-ca.pem"
ssl_certificate_verification => true
index => "%{[@metadata][target_index]}"

2

Hi @SteveParker,

this is similar to what I do. I only use if statements, no else if or else. I have a fall back value in case my version of %{[@metadata][target_index]}

Here's a couple of my filters

  if [event][module] {
    mutate {
      copy => {
       "[event][module]" => "[@metadata][index]"
      }
    }
    alter {
      add_field => {
        "[@metadata][log_prefix]" => "dc"
      }
    }
  }

I usually set extra fields in Filebeat inputs (you could do the same in winlogbeat).

  if [@metadata][beat] {
    # Adding @metadata needed for index sharding to Filebeat logs
    mutate {
      copy => {
       "[fields][log_prefix]" => "[@metadata][log_prefix]"
       "[fields][log_idx]" => "[@metadata][index]"
      }
    }
  }

Hope that makes sense :slight_smile:

And my output looks like

output {
  elasticsearch {
        hosts => ["10.1.1.1:9200"]
        index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
  }
}
3

Thank you, I will give them a try!

4

If agent is an object that contains a type field then in logstash that is called [agent][type]

5

One more thing @SteveParker.

I think all beats already adds the @metadata you need for what you want to do. So you do not need those filters.

E.g. for Metricbeat gives you this out of the box

{
    ...
    "@metadata": { 
      "beat": "metricbeat", 
      "version": "7.6.2" 
    }
}

And from the same page the suggested output config

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
}
6

Thanks A_B, I have removed the filters and used the beats @metadata to create the index names. Much cleaner config and the result I was looking for, Cheers!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.