Grok syslog-ng stats

Feb 20 00:30:00 host syslog-ng[12336]: Log statistics; queued='global(scratch_buffers_count)=0',  processed='destination(ntpd_clients)=59', processed='destination(allusers)=0', processed='center(received)=372450', processed='destination(named_clients)=32', processed='source(local)=103306', processed='destination(periodic)=1230', processed='destination(periodic_clients)=3028', processed='destination(cron)=6829',......

I have syslog-ng stats in the above format and would like to parse each stat. For example, queued='global(scratch_buffers_count)=0' should be grokked as

statistic_type: queued
object_type: global
object_name: scratch_buffers_count
total_messages: 0

and so on for each stat.

I can only think of kv filter but that only do queued: 'global(scratch_buffers_count)=0' and there will be duplicate keys. How do I approach this?

It is unclear what you mean by that. Do you want to end up with an array of objects each of which has those 4 properties?

I would be tempted to use grok or dissect to split at the semicolon, then mutate/split and process each array entry with a ruby filter to do this.

Thanks a lot. I was able to get the stats into an array using a filter like this. How do I process each array entry with a ruby filter? Do I have to loop over the items and extract fields or is there a grok filter for that?

filter {
  grok {
    match => { "message" => ["%{SYSLOGTIMESTAMP:[syslog_timestamp]} %{SYSLOGHOST:[syslog_hostname]} %{DATA:[syslog_program]}(?:\[%{POSINT:[pid]}\])?: Log statistics; %{GREEDYDATA:[statistics]}",
                             "%{SYSLOGTIMESTAMP:[syslog_timestamp]} %{SYSLOGHOST:[syslog_hostname]} %{DATA:[syslog_program]}(?:\[%{POSINT:[pid]}\])?: %{GREEDYDATA:[syslog_message]}"] }
    date {
        match => [ "[syslog_timestamp]", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
  mutate {
    split => { "statistics" => "," }


Yes, something like this...

  mutate { split => { "stats" => "," } }
  ruby {
    code => '
      stats =
      event.get("stats").each { |x|
        x = x.strip
        x = x.split(/[=\'\)\(]+/)
        h = { "statistic_type" => x[0], "object_type" => x[1], "object_name" => x[2], "total_messages" => x[3] }
      event.set("statistics", stats)
#  mutate { remove_field => [ "stats" ] }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.