How to access beats data in logstash

Elastic Stack Logstash
1

Hi there,

I started learning Elastic stack a few weeks ago by using beats + logstash + statsd (because I already have an existing graphite/Grafana stack).

I'm stuck with metricbeats and heartbeats because I'm not able to send dynamic data to statsd.
For example (heartbeats) :

I want to increment statsd for each "up" status.
Here is heartbeat debug output:

{"@timestamp":"2017-05-27T14:43:37.959Z","beat":{"hostname":"stats","name":"stats","version":"5.4.0"},"duration":{"us":37852},"host":"www.google.fr:80","ip":"91.121.101.165","monitor":"tcp-plain@www.google.fr:80","port":"80","resolve_rtt":{"us":2453},"scheme":"tcp","tcp_connect_rtt":{"us":35275},"type":"tcp","up":true}

Here is my logstash config file:

input { 
          beats {
            port => 5044
          }
          file {
            type => "nginx-access"
            path => ["/var/log/nginx_web/default_access.log"]
          }
          heartbeat {
            interval => 10
            type => "heartbeat"
          }
        }
   filter {
          if [type] == "nginx-access" {
            grok {
              match => { "message" => "%{COMBINEDAPACHELOG}" }
            }
          ...
        }
  output {
          if [type] == "heartbeat" {
            statsd {
              host => "127.0.0.1"
              port => 8125
                increment => "%{host}.heartbeats.%{up}"
              }
          }
        if [type] == "nginx-access" {
            statsd {
              host => "127.0.0.1"
              port => 8125
                increment => "%{host}.nginx_web.response.%{response}"
              }
          }
        ...
        }

I want to be able to access "up" data in logstash and increment statsd like i do for nginx response.
But I didn't find a way to access heartbeat data.
I found, using the file output in logstash, that the output is:

{"@timestamp":"2017-05-27T15:57:30.896Z","host":"stats","@version":"1","message":"ok","type":"heartbeat"}

This message seems to be only heartbeat status (message = ok even when target is not reachable)
So I suppose I need to do something in the filter section, but i don't know what.

Thank you for your help !

Ludo

2

You're mixing two different kinds of heartbeats. The events emitted by Beats contains an up field (the first example you gave) while the heartbeat events emitted by the heartbeat input (the second example) doesn't.

3

Thank you Magnus for your answer.

I manage to access heartbeat data by filtering "tcp" type on output.
Is there a better way to target heartbeat?

tx !

4

This topic was automatically closed after 21 days. New replies are no longer allowed.