Logstash/Beats tagging not working?

Elastic Stack Logstash
1

Hi,

I'm attempting to add tags via beats input into logstash using:

input {
        beats {
                host => "192.168.1.20"
                port => 5045
                client_inactivity_timeout => 0
                tags => ["malware"]
                }
        }

filter {
if "malware" in [tags] {
    grok {
#               break_on_match => false
            patterns_dir => "/etc/logstash/conf.d/patterns/mypatterns"
            match => {
                    "message" => [
                                    "%{DATESTAMP} %{WORD:zone} - %{LOGLEVEL:loglevel} - Total time taken: %{NOTSPACE:crontimetaken} \(M:S\)",
                                    "%{DATESTAMP} %{WORD:zone} - %{LOGLEVEL:loglevel} - %{DATA:returnstatustext} %{INT:returnstatuscode}"
                                 ]
                     }
         }

    mutate {
            convert => {
                            "[returnstatuscode]" => "integer"
                       }
           }
   }
}

output {
if "malware" in [tags] {
    elasticsearch {
                    hosts => "192.168.1.20:9200"
                    manage_template => true
                    template_overwrite => true
                    index => "test-index-%{+YYYY.MM.dd}"
                  }
                    }
}

However it doesn't seem to pass any records into elasticsearch with this config, if I remove the tags section it passes the records in fine (obviously just without the tags).

I've also attempted to do this in filebeat using:

filebeat.prospectors:

- type: log
  enabled: true
  tags: [ "malware" ]
  paths:
     - /var/log/test.log

But again, it doesn't seem to pass the tag for me to search in Kibana.

Does anyone have any ideas as to why this may be?

2

Temporarily replace the elasticsearch output with a stdout { codec => rubydebug } output while you're debugging.

3

Hey Magnus,

Thanks for the quick reply.

I've enabled this setting now and I'm not seeing much in the logs that could help:

Feb  1 14:43:45 monitoring logstash[14733]:           "prospector" => {
Feb  1 14:43:45 monitoring logstash[14733]:         "type" => "log"
Feb  1 14:43:45 monitoring logstash[14733]:     },
Feb  1 14:43:45 monitoring logstash[14733]:             "loglevel" => "NOTICE",
Feb  1 14:43:45 monitoring logstash[14733]:     "returnstatustext" => "Cron finished with return status",
Feb  1 14:43:45 monitoring logstash[14733]:           "@timestamp" => 2018-02-01T14:43:44.238Z,
Feb  1 14:43:45 monitoring logstash[14733]:                 "zone" => "GMT",
Feb  1 14:43:45 monitoring logstash[14733]:               "source" => "/tmp/cron-test.log",
Feb  1 14:43:45 monitoring logstash[14733]:              "message" => "01-02-2018 07:15:08 GMT - NOTICE - Cron finished with return status 0.",
Feb  1 14:43:45 monitoring logstash[14733]:     "returnstatuscode" => 0,
Feb  1 14:43:45 monitoring logstash[14733]:                 "host" => "test-elk"
Feb  1 14:43:45 monitoring logstash[14733]: }
4

Please show the full message. Specifically, I'm interested in the presence of a tags field. The fact that the grok filter conditioned on a tag to be present obviously worked so I really would expect a tags field here.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.