Get %{[@metadata][beat]}-%{[@metadata][version]} index

Elastic Stack Beats
1

I upgraded from 5.6.3 to 6.0 and i have two question:

  1. Need upgrade all beat to 6.0 ? v5.6.x conflict with ES 6.0.0?
  2. I got %{[@metadata][beat]}-%{[@metadata][version]}-2017-37 in index name.
    Flow data : beat -> LS forwarder -> RabbitMQ -> LS Proccessor -> ES.

Config in LS forwarder

input {
  beats {
    port => 5044
  }
}
filter {
  if [type] == "wineventlog" and [event_id] == 5156 {
    drop { }
  }
  mutate {
   add_field => {"beatname" => "%{[@metadata][beat]}"}
   add_field => {"beattype" => "%{[@metadata][type]}"}
   add_field => {"beatversion" => "%{[@metadata][version]}"}
  }
}

output {
    rabbitmq {
        exchange => "logstash"
        exchange_type => "direct"
        key => "logstash-key"
        host => "10.1.100.10"
        vhost => "elastic-stack"
        durable => true
        persistent => true
        port => 5677
        user => "logstash"
        password => "password"

    }
}

Confing LS Proccessor

input {
    rabbitmq {
        host => "10.1.100.10"
        queue => "logstash-queue"
        key => "logstash-key"
        exchange => "logstash"
        threads => 5
        exclusive => false
        prefetch_count => 256
        vhost => "elastic-stack"
        port => 5677
        user => "logstash"
        password => "password"
    }

}

output {
    elasticsearch {
        hosts => ["http://node1:9200", "node2:9200", "node3:9200"]
        user => "user"
        password => "passs"
        manage_template => false
        index => "%{beatname}-%{beatversion}-%{+xxxx.ww}"
        #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+xxxx.ww}"
        document_type => "%{beattype}"
    }
}

Somehing wrong in my config? . This config is worked with 5.6.x. i just add more field beatversion for 6.0

2

Need upgrade all beat to 6.0 ? v5.6.x conflict with ES 6.0.0?

There are some minor differences to the events schema. That's why the template mapping and index names are versioned. Still kibana uses index mapping <beatname>-*. Beats 5.6 and 6.0 both work with Elasticsearch 6.0. But if possible ensure all beats using the same version.

No idea about your logstash config. For debugging I would add stdout { codec => rubydebug } to the output section, so I can inspect the actual events.

3

Yeah, when i upgrade success ES and LS to 6.0 an run i got error about mapping index (in log of logstash)
So if dont upgrade beat version, i must change index name to <beat name>-<beat-version>-xxxx.ww right?

Thanks for support!

4

Yeah, you should include the beat version in the index name.

5

Yeah, i solved my problem.
In my case, i have both of beat 5.4 and 6.0. So i cant use [@metadata][version] because in beat 5.x, dont have this field.
So i used [beat][vesion]. It worked with beat version 5.4 and 6.0.

6

This topic was automatically closed after 21 days. New replies are no longer allowed.