Grok syslog-ng stats

[pdna (pdna356)]

Feb 20 00:30:00 host syslog-ng[12336]: Log statistics; queued='global(scratch_buffers_count)=0',  processed='destination(ntpd_clients)=59', processed='destination(allusers)=0', processed='center(received)=372450', processed='destination(named_clients)=32', processed='source(local)=103306', processed='destination(periodic)=1230', processed='destination(periodic_clients)=3028', processed='destination(cron)=6829',......

I have syslog-ng stats in the above format and would like to parse each stat. For example, queued='global(scratch_buffers_count)=0' should be grokked as

statistic_type: queued
object_type: global
object_name: scratch_buffers_count
total_messages: 0

and so on for each stat.

I can only think of kv filter but that only do queued: 'global(scratch_buffers_count)=0' and there will be duplicate keys. How do I approach this?

[Badger]

@pdna

and so on for each stat

It is unclear what you mean by that. Do you want to end up with an array of objects each of which has those 4 properties?

I would be tempted to use grok or dissect to split at the semicolon, then mutate/split and process each array entry with a ruby filter to do this.

[pdna (pdna356)]

Thanks a lot. I was able to get the stats into an array using a filter like this. How do I process each array entry with a ruby filter? Do I have to loop over the items and extract fields or is there a grok filter for that?

filter {
  grok {
    match => { "message" => ["%{SYSLOGTIMESTAMP:[syslog_timestamp]} %{SYSLOGHOST:[syslog_hostname]} %{DATA:[syslog_program]}(?:\[%{POSINT:[pid]}\])?: Log statistics; %{GREEDYDATA:[statistics]}",
                             "%{SYSLOGTIMESTAMP:[syslog_timestamp]} %{SYSLOGHOST:[syslog_hostname]} %{DATA:[syslog_program]}(?:\[%{POSINT:[pid]}\])?: %{GREEDYDATA:[syslog_message]}"] }
      }
    date {
        match => [ "[syslog_timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
  mutate {
    split => { "statistics" => "," }
    }

}

[Badger]

@pdna

How do I process each array entry with a ruby filter? Do I have to loop over the items and extract fields

Yes, something like this...

  mutate { split => { "stats" => "," } }
  ruby {
    code => '
      stats = Array.new
      event.get("stats").each { |x|
        x = x.strip
        x = x.split(/[=\'\)\(]+/)
        h = { "statistic_type" => x[0], "object_type" => x[1], "object_name" => x[2], "total_messages" => x[3] }
        stats.push(h)
      }
      event.set("statistics", stats)
    '
  }
#  mutate { remove_field => [ "stats" ] }