Logstash pipelines: "if [fileset][module]" sometimes not working properly

MartijnS · December 15, 2018, 12:35pm

Hello,

Since a week or so I'm trying Elastic to determine wether or not it's something we can use at work (webhosting company). I've got it all setup (filebeat and metricbeat, Logstash, Elastic and Kibana). Within Logstash I've setup a multiple pipelines. If it's a "filebeat" input, it will check if the fileset.module == apache. If so, it will be send to the apache pipeline. Works like a charm! For today, there are already +- 11.680 documents inserted in corresponding index. This also works for fileset.module == system, etc. Anything else will be send to the filebeat-unsorted-$date index. This also works. However, in the "filebeat-unsorted-$date" index are also some entries that do have the "fileset.module" set to Apache. I don't understand (yet) why they are being processed to the unsorted index instead of the Apache index. This also happens for the system and nginx fileset.modules. The records showing up in the unsorted index, does not appear in the apache index.

Anyone can give me a clue on how this is possible? The related configurations and locations are:
pipelines.yml
/etc/logstash/pipelines.yml

beats.conf: this is the config file that determines if the input is a metricbeat or filebeat entry. Not yet sure if this is the way to go or to split it up and give each type it's own input, etc.
/etc/logstash/conf.d/beats.yml

filebeat.conf: handles all filebeat related documents/messages. This is also the configuration that sends Apache documents to the apache pipeline, etc.:
/etc/logstash/conf.d/filebeat.conf

apache_pipeline.conf: handles the apache documents, sends them to the according index in Elasticsearch.
/etc/logstash/conf.d/filebeat/apache_pipeline.conf
filebeat_pipeline.conf: handles the "unsorted" documents / documents that couldn't be matched in the filebeat.conf file
/etc/logstash/conf.d/filebeat/filebeat_pipeline.conf

I've placed the configurations on gist.github.com:

gist.github.com

https://gist.github.com/spiek90/5a8c92396b2897f700f01c21d51d1c79

apache_pipeline.conf

input {
  pipeline {
    address => "filebeat_apache_pipeline"
  }
}
filter {
  if [fileset][name] == "access" {
    grok {
      #match => { "message" => "%{COMBINEDAPACHELOG}"}
      match => { "message" => ["%{IPORHOST:[apache][remote_hostname]} - %{DATA:[apache][remote_user]} \[%{HTTPDATE:[apache][request][time]}\] \"%{WORD:[apache][request][method]} %{DATA:[apache][request][url]} HTTP/%{NUMBER:[apache][request][protocol]}\" %{NUMBER:[apache][status_final]} (%{NUMBER:[apache][response][bytes_clf]}|-)( \"%{DATA:[apache][referer]}\")?( \"%{DATA:[apache][user_agent]}\")?"] }

This file has been truncated. show original

beats.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate_authorities => ["/etc/logstash/ssl/ca.crt"]
    ssl_certificate => "/etc/logstash/ssl/<domain.tld>.crt"
    ssl_key => "/etc/logstash/ssl/<domain.tld>.p8.key"
    ssl_verify_mode => "force_peer"
  }
}

This file has been truncated. show original

filebeat.conf

input {
  pipeline {
    address => "filebeat"
  }
}
filter {
  mutate {
   remove_field => ["beatname"]
  }
}

This file has been truncated. show original

There are more than three files. show original

If there is anything else you'd need to know, please let me know! Hope someones see what's missing in this configuration or where I messed up!

system · January 12, 2019, 12:35pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.