# Lost All Indices After Adding Additional Pipelines

**URL:** https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858
**Category:** Logstash
**Created:** [June 24, 2021, 1:29am UTC](https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858 "2021-06-24T01:29:02Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Steve\_K](https://avatars.discourse-cdn.com/v4/letter/s/3ec8ea/32.png) [@Steve\_K](https://discuss.elastic.co/u/Steve_K)
#### Post date: [June 24, 2021, 1:29am UTC](https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858/1 "2021-06-24T01:29:03Z")

</div>

So after I cleared all of the indices and added an additional method to my pipeline configuration, my indices will not repopulate.

I have added my code below for the original configuration and then reconfiguration.

Original Configuration:

```auto
input {
  beats {
    port => 5044
  }
}

filter {
grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:dst_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?<status>\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:src_host}.*" }
    add_field => { "activity" => "SSH Logins" }
    add_tag => "linux_auth"
    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
   }
}

```

Reconfiguration:

```auto
input {
  beats {
    port => 5044
    type => "beats"
    tags => ["beats"]
  }
  udp {
    port => 514
    type => "syslog"
    tags => ["syslog"]
  }
}

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
    date {
       match => ["timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss"]
    }
  }
 if [type] == "beats" {
    match => { "message" => "%{SYSLOGTIMESTAMP:timestamp}\s+%{IPORHOST:dst_host}\s+%{WORD:syslog_program}\[\d+\]:\s+(?<status>\w+\s+password)\s+for\s+%{USER:auth_user}\s+from\s+%{SYSLOGHOST:src_host}.*" }
    add_field => { "activity" => "SSH Logins" }
    add_tag => "linux_auth"
  }
}

output {
  if "beats" in [tags] {
   elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
   }
  }
  if "syslog" in [tags] {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "syslogs-%{+YYYY.MM.dd}"
   }
 }
}

```

After I had restarted logstash, only syslog port is open 514 and not beats port 5044.

What could be the issue?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 24, 2021, 4:56am UTC](https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858/2 "2021-06-24T04:56:32Z")

</div>

Welcome to our community! 😃

> [@Steve\_K](#):
>
> So after I cleared all of the indices

Can you elaborate more on what you did here?

Also, what do your Logstash logs show is happening?

---

<div class="post-metadata">

### Author: ![Steve\_K](https://avatars.discourse-cdn.com/v4/letter/s/3ec8ea/32.png) [@Steve\_K](https://discuss.elastic.co/u/Steve_K)
#### Post date: [June 24, 2021, 1:58pm UTC](https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858/3 "2021-06-24T13:58:08Z")

</div>

I followed this guy's post and my indices are now working.

> [@Using two input plugins ( beats and udp ) and output it to Elasticsearch](https://discuss.elastic.co/t/using-two-input-plugins-beats-and-udp-and-output-it-to-elasticsearch/92097/5):
>
> Thanks Paris. I am glad you replied asap. I really appreciate your help. Thanks a lot !! You made my Day !! Thanks you very much !! The Beats shipper automatically sets the type field on the event. You cannot override this setting in the Logstash config. If you specify a setting for the type config option in Logstash, it is ignored. It helped me a lot to resolve the above issue. I haven't changed the filter configurations. I have used the following configurations and it resolved the ab…

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 22, 2021, 3:04pm UTC](https://discuss.elastic.co/t/lost-all-indices-after-adding-additional-pipelines/276858/5 "2021-07-22T15:04:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.