# Duplicate logs when using out\_elasticsearch

**URL:** <https://discuss.elastic.co/t/duplicate-logs-when-using-out-elasticsearch/91537>\
**Category:** Elasticsearch\
**Created:** [July 2, 2017, 6:07am UTC](https://discuss.elastic.co/t/duplicate-logs-when-using-out-elasticsearch/91537 "2017-07-02T06:07:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jeremy\_Woo](https://avatars.discourse-cdn.com/v4/letter/j/58956e/32.png) [@Jeremy\_Woo](https://discuss.elastic.co/u/Jeremy_Woo)\
**Post date:** [July 2, 2017, 6:07am UTC](https://discuss.elastic.co/t/duplicate-logs-when-using-out-elasticsearch/91537/1 "2017-07-02T06:07:57Z")

</div>

hi, guys

we use `ELK` to analysis logs, recently we find that if we use the `elasticsearch` output plugin, the total amount of logs is double, but if we use the `amazon_es` output plugin, the total amount of logs is correct.  
our configuration looks like:

```auto
input {
    file {
        path => "/data/web/test/*.info.log"
        type => "info"
        tags => ["old","new"]
    }
}
filter {
    grok {
        patterns_dir => ["/etc/logstash/patterns", "/opt/logstash/patterns", "/opt/logstash/extra_patterns"]
        match => { "message" => "%{TIMESTAMP_ISO8601:logtime}%{TK_TAB}%{GREEDYDATA:heading}%{TK_TAB}%{GREEDYDATA:data}"}
        remove_field => ["message"]
    }
    date {
        match => ["logtime", "yyyy-MM-dd'T'HH:mm:ssZZ"]
        remove_field => ["logtime"]
    }
    json {
        source => "data"
        target => "@fields"
        remove_field => ["data"]
    }

    metrics {
        meter => "events"
        add_tag => "metric"
    }
}
output {
    if "new" in [tags] {
        amazon_es {
            hosts => ["my_aws_es.amazonaws.com"]
            region => "my_es_region"
            aws_access_key_id => 'my_key_id'
            aws_secret_access_key => 'my_key_secret'
            index => "aws_es_test_%{+YYYY.MM.dd}"
            document_type => "%{type}"
        }
    }

    if "old" in [tags] {
         elasticsearch {
            hosts => ["127.0.0.1:9200"]
            index => "es_test_%{+YYYY.MM.dd}"
            document_type => "%{type}"
        }
    }
}

```

On `kibana`, we find that the total amount of logs which sent to the local `elasticsearch` is double of the logs which sent to `aws elasticsearch`

Can anyone tell me why could this happen? any answers will be appreciated

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2017, 6:08am UTC](https://discuss.elastic.co/t/duplicate-logs-when-using-out-elasticsearch/91537/2 "2017-07-30T06:08:14Z")

</div>

Closed on July 30, 2017, 6:08am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.