# Duplicate content in indexes

**URL:** <https://discuss.elastic.co/t/duplicate-content-in-indexes/131821>\
**Category:** Logstash\
**Created:** [May 14, 2018, 11:23pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821 "2018-05-14T23:23:23Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![xternal](https://avatars.discourse-cdn.com/v4/letter/x/c4cdca/32.png) [@xternal](https://discuss.elastic.co/u/xternal)\
**Post date:** [May 14, 2018, 11:23pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/1 "2018-05-14T23:23:23Z")

</div>

Hi,  
I was hoping someone might be able to help me with this logstash.conf. Everything from beats is ending up in the wazuh index and the winlogbeat index. However the wazuh data does not end up in the winglogbeat index. If I comment out the beats input, I just get the wazuh data.

Many Thanks!

```
`input {
   file {
       type => "wazuh-alerts"
       path => "/var/ossec/logs/alerts/alerts.json"
       codec => "json"
   }
   beats {
       port => 5044
       ssl => true
       ssl_certificate => "/etc/pki/tls/certs/logstash.cer"
       ssl_key => "/etc/pki/tls/private/logstash.key"
  }

}

filter {
    if [type] == "wazuh-alerts" {
      if [data][srcip] {
          mutate {
              add_field => ["@src_ip", "%{[data][srcip]}" ]
          }
      }
      if [data][aws][sourceIPAddress] {
          mutate {
              add_field => ["@src_ip", "%{[data][aws][sourceIPAddress]}" ]
          }
      }
    }
}

filter {
    if [type] == "wazuh-alerts" {
      geoip {
          source => "@src_ip"
          target => "GeoLocation"
          fields => ["city_name", "continent_code", "country_code2", "country_name", "region_name", "location"]
      }
      date {
          match => ["timestamp", "ISO8601"]
          target => "@timestamp"
      }
      mutate {
          remove_field => ["timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type","@src_ip"]
      }
    }
}

output {
  if [type] == "wazuh-alerts" {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
        document_type => "wazuh"
    }
  }
  else {
    elasticsearch {
      hosts => ["localhost:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
}

```

`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2018, 12:09am UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/2 "2018-05-15T00:09:50Z")

</div>

Is it possible that you are pointing -f to a directory that contains more than one configuration file (e.g. the one you want and a backup copy that has another output)? How are you starting logstash?

---

<div class="post-metadata">

**Author:** ![xternal](https://avatars.discourse-cdn.com/v4/letter/x/c4cdca/32.png) [@xternal](https://discuss.elastic.co/u/xternal)\
**Post date:** [May 15, 2018, 1:05am UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/3 "2018-05-15T01:05:27Z")

</div>

No it is just the logstash.conf. Also it stops if I comment the beats line out.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 15, 2018, 1:42pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/4 "2018-05-15T13:42:12Z")

</div>

It's really hard to believe the events are flowing through both the if and else code blocks. When the beats events are in the wazuh index what document type do they have?

---

<div class="post-metadata">

**Author:** ![motts](https://avatars.discourse-cdn.com/v4/letter/m/839c29/32.png) [@motts](https://discuss.elastic.co/u/motts)\
**Post date:** [May 15, 2018, 9:00pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/5 "2018-05-15T21:00:15Z")

</div>

I had this issue myself. Turns out, there was an error in the Kibana index in Elasticseaerch. I got rid of that Kibana index and data was going to the correct indices.

Make sure the only files in /logstash/logstash.conf directory are only .conf files for logstash as well.

See here:

> [@Data is sent to created index and the default index](https://discuss.elastic.co/t/data-is-sent-to-created-index-and-the-default-index/113845):
>
> I have an interesting issue where data is being sent to the new index and also the old default index. I am using Elasticsearch 2.4 and here is my output file: output { if [type] == "Corporate" { elasticsearch { hosts =\> ["localhost:9200"] index =\> "corp\_windows\_events-%{+YYYY-MM-dd}" } } else { elasticsearch { hosts =\> ["localhost:9200"] } } } I checked both indic…

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 12, 2018, 9:00pm UTC](https://discuss.elastic.co/t/duplicate-content-in-indexes/131821/6 "2018-06-12T21:00:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.