Logstash not indexing properly

Elastic Stack Logstash
1

Hi,

I have a single-node Elastic stack running with the following architecture:
Filebeat > Logstash > Elasticsearch

Filebeat is doing the tagging and logstash will point the documents to be indexed on different indices depending on the tags.

However, I noticed within Kibana "Discover" tab, documents are indexed in the correct indices. At the same time, it is also indexed into index named "indexfordrop-2" (this is meant as a "catch-all" document and also for troubleshooting this issue). For example, I have Cisco logs indexed into "filebeat-security-ciscoasa-%{+YYYY.MM.dd}" and at the same time, it is also indexed into "indexfordrop-2".

This is really puzzling, why would logstash index it to 2 different indices.

Here's what I gathered so far:

  1. The documents on both indices seems to be the same, I randomly searched through 5-10 documents and both incides has the same document information (but with different document ID of course).
  2. This only happens for logs that is enabled on filebeat modules (Cisco in this case).

Any idea why this is happening?

Logstash conf.d output:

output {
  if "extsyslog" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "filebeat-security-extsyslog-2"
    pipeline => "extsyslogpipeline"
    }
  }

  if "customintel-csv" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "siem-custom-intel"
    pipeline => "customintel"
    }
   }

  if "cisco-asa" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "filebeat-security-ciscoasa-%{+YYYY.MM.dd}"
    pipeline => "%{[@metadata][pipeline]}"
    }
   }

else {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "indexfordrop-2"
    pipeline => "%{[@metadata][pipeline]}"
    }
  }
}
2

The structure of your output section is

output {
  if "extsyslog" in [tags] { elasticsearch {} }
  if "customintel-csv" in [tags] { elasticsearch {} }
  if "cisco-asa" in [tags] { 
    elasticsearch {}
   } else {
    elasticsearch {}
  }
}

If events only ever have one of those tags I would expect everything except events with the cisco-asa tag to go to the fourth Elasticsearch output.

It is hard to believe that an event could be going to both the third and fourth outputs.

3

Yes, that's what I thought too. It's really puzzling me.

I attached some screenshots here from both indices.

From "filebeat-security-ciscoasa-%{+YYYY.MM.dd}"

Filebeat
Filebeat3752×978 95.7 KB

From "indexfordrop-2"

Indexfordrop
Indexfordrop3768×913 86.2 KB

Notice the logs are identical. Not sure if this is a bug, or its just the way I write my logstash output.

Edit: Hang on, now that I re-read your reply:

  1. Are you saying, if tags with for example "extsyslog" tags, it will still go to first and the fourth Elasticsearch output?
  2. There is another statement in logstash that was added few days ago below the cisco-asa. Here's the updated logstash output:
output {
  if "extsyslog" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "filebeat-security-extsyslog-2"
    pipeline => "extsyslogpipeline"
    }
  }

  if "customintel-csv" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "siem-custom-intel"
    pipeline => "customintel"
    }
   }

  if "cisco-asa" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "filebeat-security-ciscoasa-%{+YYYY.MM.dd}"
    pipeline => "%{[@metadata][pipeline]}"
    }
   }

  if "threatintel-malwarebazaar" in [tags] {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "filebeat-security-threatintel-2"
    pipeline => "%{[@metadata][pipeline]}"
    }
   }
   
else {
    elasticsearch {
    ssl => true
    ssl_certificate_verification => false
    cacert => "/etc/logstash/elasticsearch-ca.pem"
    hosts => "https://10.20.14.232:9200"
    user => "${LS_USER}"
    password => "${LS_PWD}"
    manage_template => true
    index => "indexfordrop-2"
    pipeline => "%{[@metadata][pipeline]}"
    }
  }
}
4

Yes, and in your updated configuration the fifth elasticsearch output will receive everything that is not tagged with "threatintel-malwarebazaar". I think what you want is

  if "extsyslog" in [tags] { elasticsearch {} }
  else if "customintel-csv" in [tags] { elasticsearch {} }
  else if "cisco-asa" in [tags] { elasticsearch {} } 
  else { elasticsearch {} }
5

Works like a charm!

Thanks for helping. Learnt something new today.

And Merry Christmas to you :slight_smile:

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.