Duplicate logs when using out_elasticsearch

Elastic Stack Elasticsearch
1

hi, guys

we use ELK to analysis logs, recently we find that if we use the elasticsearch output plugin, the total amount of logs is double, but if we use the amazon_es output plugin, the total amount of logs is correct.
our configuration looks like:

input {
    file {
        path => "/data/web/test/*.info.log"
        type => "info"
        tags => ["old","new"]
    }
}
filter {
    grok {
        patterns_dir => ["/etc/logstash/patterns", "/opt/logstash/patterns", "/opt/logstash/extra_patterns"]
        match => { "message" => "%{TIMESTAMP_ISO8601:logtime}%{TK_TAB}%{GREEDYDATA:heading}%{TK_TAB}%{GREEDYDATA:data}"}
        remove_field => ["message"]
    }
    date {
        match => [ "logtime", "yyyy-MM-dd'T'HH:mm:ssZZ" ]
        remove_field => ["logtime"]
    }
    json {
        source => "data"
        target => "@fields"
        remove_field => ["data"]
    }

    metrics {
        meter => "events"
        add_tag => "metric"
    }
}
output {
    if "new" in [tags] {
        amazon_es {
            hosts => ["my_aws_es.amazonaws.com"]
            region => "my_es_region"
            aws_access_key_id => 'my_key_id'
            aws_secret_access_key => 'my_key_secret'
            index => "aws_es_test_%{+YYYY.MM.dd}"
            document_type => "%{type}"
        }
    }

    if "old" in [tags] {
         elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "es_test_%{+YYYY.MM.dd}"
            document_type => "%{type}"
        }
    }
}

On kibana, we find that the total amount of logs which sent to the local elasticsearch is double of the logs which sent to aws elasticsearch

Can anyone tell me why could this happen? any answers will be appreciated

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.