# Logstash not indexing properly **URL:** \ **Category:** Logstash\ **Created:** [December 24, 2021, 1:23am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853 "2021-12-24T01:23:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![eleong](https://avatars.discourse-cdn.com/v4/letter/e/7cd45c/32.png) [@eleong](https://discuss.elastic.co/u/eleong)\ **Post date:** [December 24, 2021, 1:23am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/1 "2021-12-24T01:23:59Z")
Hi, I have a single-node Elastic stack running with the following architecture: Filebeat \> Logstash \> Elasticsearch Filebeat is doing the tagging and logstash will point the documents to be indexed on different indices depending on the tags. However, I noticed within Kibana "Discover" tab, documents are indexed in the correct indices. At the same time, it is also indexed into index named "indexfordrop-2" (this is meant as a "catch-all" document and also for troubleshooting this issue). For example, I have Cisco logs indexed into "filebeat-security-ciscoasa-%{+YYYY.MM.dd}" and at the same time, it is also indexed into "indexfordrop-2". This is really puzzling, why would logstash index it to 2 different indices. Here's what I gathered so far: 1. The documents on both indices seems to be the same, I randomly searched through 5-10 documents and both incides has the same document information (but with different document ID of course). 2. This only happens for logs that is enabled on filebeat modules (Cisco in this case). Any idea why this is happening? Logstash conf.d output: ```auto output { if "extsyslog" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "filebeat-security-extsyslog-2" pipeline => "extsyslogpipeline" } } if "customintel-csv" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "siem-custom-intel" pipeline => "customintel" } } if "cisco-asa" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "filebeat-security-ciscoasa-%{+YYYY.MM.dd}" pipeline => "%{[@metadata][pipeline]}" } } else { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "indexfordrop-2" pipeline => "%{[@metadata][pipeline]}" } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 24, 2021, 2:36am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/2 "2021-12-24T02:36:31Z")
The structure of your output section is ```auto output { if "extsyslog" in [tags] { elasticsearch {} } if "customintel-csv" in [tags] { elasticsearch {} } if "cisco-asa" in [tags] { elasticsearch {} } else { elasticsearch {} } } ``` If events only ever have one of those tags I would expect everything except events with the cisco-asa tag to go to the fourth Elasticsearch output. It is hard to believe that an event could be going to both the third and fourth outputs. ---
**Author:** ![eleong](https://avatars.discourse-cdn.com/v4/letter/e/7cd45c/32.png) [@eleong](https://discuss.elastic.co/u/eleong)\ **Post date:** [December 24, 2021, 7:50am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/3 "2021-12-24T07:50:49Z")
Yes, that's what I thought too. It's really puzzling me. I attached some screenshots here from both indices. From "filebeat-security-ciscoasa-%{+YYYY.MM.dd}" ![Filebeat](https://us1.discourse-cdn.com/elastic/original/3X/b/0/b08afb8b950ed37c7214188aa416438f1b77e35c.png) From "indexfordrop-2" ![Indexfordrop](https://us1.discourse-cdn.com/elastic/original/3X/f/f/ff3f18f71a5a398e078fb7ab4997adf301de0105.png) Notice the logs are identical. Not sure if this is a bug, or its just the way I write my logstash output. Edit: Hang on, now that I re-read your reply: 1. Are you saying, if tags with for example "extsyslog" tags, it will still go to first **and** the fourth Elasticsearch output? 2. There is another statement in logstash that was added few days ago below the cisco-asa. Here's the updated logstash output: ```auto output { if "extsyslog" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "filebeat-security-extsyslog-2" pipeline => "extsyslogpipeline" } } if "customintel-csv" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "siem-custom-intel" pipeline => "customintel" } } if "cisco-asa" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "filebeat-security-ciscoasa-%{+YYYY.MM.dd}" pipeline => "%{[@metadata][pipeline]}" } } if "threatintel-malwarebazaar" in [tags] { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "filebeat-security-threatintel-2" pipeline => "%{[@metadata][pipeline]}" } } else { elasticsearch { ssl => true ssl_certificate_verification => false cacert => "/etc/logstash/elasticsearch-ca.pem" hosts => "https://10.20.14.232:9200" user => "${LS_USER}" password => "${LS_PWD}" manage_template => true index => "indexfordrop-2" pipeline => "%{[@metadata][pipeline]}" } } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [December 24, 2021, 4:59pm UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/4 "2021-12-24T16:59:47Z")
> [@eleong](#): > > Are you saying, if tags with for example "extsyslog" tags, it will still go to first **and** the fourth Elasticsearch output? Yes, and in your updated configuration the fifth elasticsearch output will receive everything that is not tagged with "threatintel-malwarebazaar". I think what you want is ```auto if "extsyslog" in [tags] { elasticsearch {} } else if "customintel-csv" in [tags] { elasticsearch {} } else if "cisco-asa" in [tags] { elasticsearch {} } else { elasticsearch {} } ``` ---
**Author:** ![eleong](https://avatars.discourse-cdn.com/v4/letter/e/7cd45c/32.png) [@eleong](https://discuss.elastic.co/u/eleong)\ **Post date:** [December 25, 2021, 3:50am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/5 "2021-12-25T03:50:07Z")
Works like a charm! Thanks for helping. Learnt something new today. And Merry Christmas to you 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 22, 2022, 3:50am UTC](https://discuss.elastic.co/t/logstash-not-indexing-properly/292853/6 "2022-01-22T03:50:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.