Topbeat with redis

Elastic Stack Beats
1

topbeat version 1.0.0rc
logstash 2.0
redis

I am using the following in my logstash conf:

input {
beats {
port => 5044
type => "X"
}
}

output {
if [type] == "X" {
redis {
host => "192.168.99.4"
data_type => "list"
key => "topbeat"
}
}
}

This is not working. If I comment out type in my input and output it works fine. Why is type not working. Am I missing something. I use a very similar config in other logstash files and it works just fine.

Thanks

2

I assume the issue here is that "type" is already used by topbeat to store data. What happens if you use instead of type for example my_type?

3

type is kind of a special field in beats and logstash. If type is already set in input event, It won't be overwritten by the configurated 'type' variable in logstash config file. See code comment documenting behavior

In case of topbeat the values of type are 'proc', 'system' and 'filesystem'. The beat its name is stored in [@metadata][beat].

Instead of 'type' you can use the 'add_field' config option to add another field with custom name to the generated event (store in [@metadata][my_name] if you don't want to store the custom field in redis). But I'd recommend not doing so, as 'type' or custom field will add same field to every event, independent of actual event type or source making it inflexible for later additions (e.g. adding another beat to the system).

Maybe something like this better fits your use-case (untested):

input {
    beats {
        port => 5044
    }
}
output {
    if [@metadata] and [@metadata][beat] {
        redis {
            host => "192.168.99.4"
            data_type => "list"
            key => "%{[@metadata][beat]}"
        }
    }
}

Now every single beat you connect to logstash will produce it's own key in redis.

4

Thank you

This config doesn't work for some reason. If I do this

input {
    beats {
        port => 5044
    }
}
output {
#    if [@metadata] and [@metadata][beat] {
        redis {
            host => "192.168.99.4"
            data_type => "list"
            key => "%{[@metadata][beat]}"
        }
#    }
}

it works but not with the if statement included but the key name is literally "%{[@metadata][beat]}". It doesn't appear that "%{[@metadata][beat]}" is being picked up as a variable.

EDIT:

It appears that when I take out the if statement topbeat is still not functioning as expected. I am seeing the key in redis but other data is in there from another source not topbeat. So I think I have a more fundamental issue of logstash maybe not reading topbeat. I do see the port open using netstat and I see topbeat running using ps aux.

Is there a topbeat error log I can check

5

Figured it out. User error (BIG TIME). I fat fingered my host in the config file.

Thanks for the help

Side question. If I spin up multiple servers with topbeat running and use your above config, how will I be able to differentiate between them in elasticsearch?

6

Ever beat sends currently under the field "shipper" the hostname or the name set in the configuration. This should make it possible to differentiate between the beats. This field will change in the next release. For more details see here: https://github.com/elastic/libbeat/issues/281

7

So I've been testing the heck out of topbeat and I'm not sure if its a bug or not but using the following:

input {
  beats {
    port => 5044
    add_field => { "[@metadata][stage]" => "topbeat_raw" }
  }
}
output {
  stdout { codec => rubydebug } metadata => true } }
}

I see my topbeat output but the new metadata field is not added (see below):

{
    "@timestamp" => "2015-11-10T13:02:15.997Z",
         "count" => 1,
          "proc" => {
          "cpu" => {
                  "user" => 0,
                "user_p" => 0,
                "system" => 0,
                 "total" => 0,
            "start_time" => "11:59"
        },
          "mem" => {
             "size" => 0,
              "rss" => 0,
            "rss_p" => 0,
            "share" => 0
        },
         "name" => "rcuob/26",
          "pid" => 83,
         "ppid" => 2,
        "state" => "sleeping"
    },
       "shipper" => "satcon99",
          "type" => "proc",
      "@version" => "1",
     "@metadata" => {
        "beat" => "topbeat",
        "type" => "proc"
    }
}

Is there a reason the add_field syntax is not working? If I add it in a filter statement with the mutate plugin it works fine but according to the beats docs I should be able to use the add_field syntax in the input section.

8

which beats plugin version have you installed? add_field syntax not working was fixed here.

You can update the plugin via:

$ bin/plugin update logstash-input-beats
9

I will try checking my version tomorrow and updating if needed. Thanks for
pointing that out