Unable to change output index according to event fields

Elastic Stack Logstash
1

I want to use different indices for "filebeat" and "metricbeat" input sources, shipping events to the logstash.

Here is the logstash.conf we are using.

input
{
    beats
    {
        ssl => false
        host => "0.0.0.0"
        port => 5044
    }
    gelf
    {
        host => "0.0.0.0"
        port => 12201
    }
    http
    {
        ssl => false
        host => "0.0.0.0"
        port => 8888
    }
    tcp
    {
        mode => "server"
        host => "0.0.0.0"
        port => 5010
    }
    udp
    {
        host => "0.0.0.0"
        port => 5000
    }
}

output
{
    if [fields][beat] == "metricbeat" {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            document_id => "%{logstash_checksum}"
            index => "%{[fields][beat]}-%{+YYYY.MM.dd}"
        }
    } else {
        elasticsearch {
            hosts => ["127.0.0.1:9200"]
            document_id => "%{logstash_checksum}"
            index => "logstash-%{+YYYY.MM.dd}"
        }
    }
}

The index is being created but only one event is inserted in the index.

The metricbeat configuration has a decicated field named beat.

fields:
    beat: metricbeat

Here is the listing of all the indices.

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2017.11.10             SCPT2qBJTsK1Hdgh3sOOCQ   5   1       8933            1      7.8mb          7.8mb
yellow open   metricbeat-2017.11.10           RGWdtyJFS76bSfeA-lLAZg   5   1          1           55     87.4kb

I'm having a hard time routing the beat events in separate indices.
Please help me figure it out.

2

How is logstash_checksum calculated? Could it be that it does not exist or always evaluates to the same thing for Metricbeat records, causing the same record to be updated over and over again?

1 Like
3

Removing this key worked, partially. Thank you!
It was present in the default configuration file that I found in my bitnami based setup.

Now the events are being indexed in the metricbeat-* index but still, some of the metricbeat generated events are sent to the logstash-* index too.

I'm afraid I might miss some events if I keep observing the dedicated index for events (metricbeat-*)

Edit:
It looks like all the metricbeat events are indexed in the logstash-* index too, can't get my head around it.

4

Show an example event. Copy/paste from the JSON tab in Kibana.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.