Parsing tags

Elastic Stack Logstash
1

How can I parse this using grok filter

      "tags" => [
    [0] "test-category",
    [1] "beats_input_raw_event"

Tried this won't work

      match => { "[tags][0]" => "%{WORD:category}" }

Thanks

2 
    grok {
        match => { "tags[0]" => "%{WORD:category}" }
    }
1 Like
3

What are you really trying to do here? Why is the name of the first tag interesting?

4

I am using

shipper name and tags feature in topbeat and filebeat, to compensate for what I was doing with decorate events. Since I am not storing kafka consumer offsets in Zookeeper anymore. And the plugins don't support topic metadata at this moment.

What I noticed though that in topbeat this feature works, but not in filebeat 5.0-alpha 3

5

I'm not quite following you, but do you really need to use grok for the tags? I think it'll be easier with existence conditionals like

if "name-of-tag" in [tags] {

to check if a tag is set. Another option is to use Metricbeat instead of Topbeat and use the fields option (also available in Filebeat) to assign structured fields instead of flat tag names. Then you can define a field like "category: foo" instead of figuring out whether a "category_foo" tag has been set.

6

Here is why:

I have this config that generates the topic in kafka:

input {
beats {
host => "0.0.0.0"
port => 5044
}

}
output {

     kafka {
      bootstrap_servers =>  "broker1.com:7000,broker2.com:7000"
      topic_id => "herd-%{[@metadata][beat]}-ev1"
      codec =>  json
     }

This will correspond to:

herd-filebeat-ev1
herd-topbeat-ev1
herd-winlogbeat-ev1

Then I read from the kafka topics using decorate events:

kafka {
zk_connect => "blah1.com:2181,blah2.com:2181"
white_list => "herd-topbeat-ev1|herd-filebeat-ev1|herd-winlogbeat-ev1"
decorate_events => true
codec => json
}
}

filter {

grok {
match => { "[kafka][topic]" => "%{WORD:platform}-%{WORD:beatsource}-%{WORD:dc}" }
}

}

My index name follows this format index => "%{beatsource}-%{+YYYY.MM.dd}" which was acquired from parsing the topic name.

Now I have upgraded kafka input/output plugins to 3.x in order to have Kafka 0.9 consumer/producer, to allow consumer offset be stored in Kafka. As a result I have lost ability to use topic metadata through decorate events. I am told that is going to be reinstated at some point. Until then, if you can recommend a very generic way of passing a field or a tag from topbeat, logstash and winlogbeat that i can use in me filter I would greatly appreciate.

This is what I have, where

[beat][name] is suppsed to come from

shipper:
name: filebeat

filter {

grok {
match => { "[beat][name]" => "%{WORD:beatsource}" }
}

}

The reason I was trying to use the shipper, was to have very general high level categorization where messages that have filebeat go to, same for topbeat, and winlogbeat .

7

Hmm, okay. Metricbeat and Filebeat support adding arbitrary fields to emitted events but Winlogbeat and Topbeat currently only support tags. Since the number of possible tags is finite I'd just use a short series of conditionals like

if "source-filebeat" in [tags] {
  ...
} else if "source-topbeat" in [tags] {
  ...
}
...

to discriminate between different kinds of messages.