# Logstash saving data into two indices (Duplicated Data)

**URL:** <https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457>\
**Category:** Logstash\
**Created:** [November 15, 2020, 11:57pm UTC](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457 "2020-11-15T23:57:41Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![gonzalo2kx](https://avatars.discourse-cdn.com/v4/letter/g/c4cdca/32.png) [@gonzalo2kx](https://discuss.elastic.co/u/gonzalo2kx)\
**Post date:** [November 15, 2020, 11:57pm UTC](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457/1 "2020-11-15T23:57:41Z")

</div>

Good day,

I am currently processing the data from filebeat's system module using Logstash. The data provided by filebeat's system module (syslog and auth) is properly parsed but unfortunately it was saving the structured documents into two indices at the same time:

E.g.  
filebeat-7.6.2-2020.11.15  
filebeat-system-7.6.2-2020.11.16

This was due to a conflict I had in the output section of logstash where I had the following statements:

```auto
output{
  if condition {output}
  if condition {output}
  else {output}
}

```

Filebeat's output configuration is enabled for Logstash only and Disabled for Elasticsearch. Also, none of the following settings are defined in filebeat's configuration:

setup.template.name:  
setup.template.pattern:  
output.elasticsearch.index:

I updated my logstash output configuration to the one stated below which solved this issue:

```auto
output {
  if [event][module] == "system" {
    elasticsearch {
      user => 'logstash_writer'
      password => 'my_dummy_password'
      hosts => ["http://elasticsearch-1:9200","http://elasticsearch-2:9200","http://elasticsearch-3:9200","http://elasticsearch-4:9200"]
      cacert => '/etc/logstash/config/certs/logstash.pem'
      index => "%{[@metadata][beat]}-%{[event][module]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #index => "divya-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
  else if [event][module] == "mysql" {
    elasticsearch {
      user => 'logstash_writer'
      password => 'my_dummy_password'
      hosts => ["http://elasticsearch-1:9200","http://elasticsearch-2:9200","http://elasticsearch-3:9200","http://elasticsearch-4:9200"]
      cacert => '/etc/logstash/config/certs/logstash.pem'
      index => "%{[@metadata][beat]}-%{[event][module]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #index => "divya-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
  else if ![event][module] {
    elasticsearch {
      user => 'logstash_writer'
      password => 'my_dummy_password'
      hosts => ["http://elasticsearch-1:9200","http://elasticsearch-2:9200","http://elasticsearch-3:9200","http://elasticsearch-4:9200"]
      cacert => '/etc/logstash/config/certs/logstash.pem'
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      #index => "divya-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
      document_type => "%{[@metadata][type]}"
    }
  }
}

```

Any suggestions on how I could improve this solution to avoid future issues?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 16, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457/2 "2020-11-16T00:16:00Z")

</div>

> [@gonzalo2kx](#):
>
> Any suggestions on how I could improve this solution to avoid future issues?

You could use [pipellines](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 14, 2020, 12:16am UTC](https://discuss.elastic.co/t/logstash-saving-data-into-two-indices-duplicated-data/255457/3 "2020-12-14T00:16:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.