What is wrong in my grok filter work for logstash?

Elastic Stack Logstash
1

I would like to create a grok filter in Logstash to set as field, a concrete part of a syslog message TTN-GW string should be that field, the sort of message that is being sent to Logstash is as follows ( it comes from a syslog)

Oct 21 09:47:12 aaaa TTN-GW[1401]: #033[90m DEBUG#033[0m Handle stat #033[90mGatewayID#033[0m=SFDEGFKKGRSDFEEE #033[90madapter#033[0m=gateway-semtech

My grok filter looks like this, I have created it modifying a syslog filter (shown below),

    filter {
      if [type] == "TTN" {
        grok {
    
          match => { "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:ttn_service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "TTN_Service", "%{ttn_service}" ]
        }
      }
    }

However, it doesn't work as expected and I don't know why, any hint?

Thanks in advance!

The following grok filter for syslog parse it properly,

    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }
2

Would somethint like this work too?

filter {
  if [type] == "TTN" {
    grok {
      match => { "message", "... TTN-GW ..." }
      add_tag => [ "TTN_Service", "TTN-GW" ]
    }
    grok {
      match => { "message", "... TTN-Node-Red ..." }
      add_tag => [ "TTN_Service", "TTN-Node-Red" ]
    }
  }
}
3

The DATE pattern doesn't match your timestamp. Why are you using it instead of SYSLOGTIMESTAMP which worked?

      add_field => [ "TTN_Service", "%{ttn_service}" ]

Why not capture straight into the TTN_Service field instead of capturing into ttn_service and copying that value to TTN_Service?

4

You were right @magnusbaeck, it was the timestamp, I updated it and now it works.

Sorry I am a bit newbie in this, how do you mean?

5

Sorry I am a bit newbie in this, how do you mean?

This is unnecessarily complicated:

grok {
  match => {
    "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:ttn_service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
  }
  add_field => [ "TTN_Service", "%{ttn_service}" ]
}

It can be replaced by this:

grok {
  match => {
    "message" => "%{DATE:date} %{SYSLOGHOST:syslog_hostname} %{TTNSERVICE:TTN_Service}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
  }
}
1 Like
6

Ok, I will try it

Just for the sake of learning, why is the first option more complicated?

Thanks!

7

Because you have an unnecessary add_field and you're creating both TTN_Service and ttn_service.

1 Like
8

Understood, thanks :smiley: