Non standard log lines

Elastic Stack Logstash
1

I'm having some trouble with logstash grok filter. Specifically matching the log to a pattern.

Say I have two log messages that looks something like this:
10.12.6.190 [Thu, 24 Sep 2015 18:17:52 GMT] 404 -
192.168.48.207 [Thu, 24 Sep 2015 18:20:12 GMT] 200 1234

(redacted info that is not needed)

Here are my issues:

  1. I haven't been able to find a DATESTAMP pattern that matches this datetime (I tried DATESTAMP_RFC2822 but kept getting grokparsefailure),
  2. Notice that if I use a grok pattern that looks like this:
    %{IPORHOST:clientip} [%{GREEDYDATA:timestamp}] %{NUMBER:response:int} %{NUMBER:bytes}
    the %{NUMBER:bytes} pattern will fail on the second log.

Here are my questions:

  1. Does anybody know of a pattern that will match this datestamp?
  2. Should I use something like %{GREEDYDATA:bytes} and try to coerce it to a number later on, or is there some conditional I can use like '%{NUMBER:bytes}|-' ? Note I'd prefer not to do a multi match here just for this one case, seems like duplicating too much. As these cases grow the combinatorics of all of the possible scenarios just seems message to have to handle in a multi match / multi pattern way.
  3. I think I've found all of the easily accessible resources like:
    https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns
    http://grokdebug.herokuapp.com/
    https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
    but I'm still not clicking with grok too well. I think what I'd really benefit from is complex example logstash configurations with the log files they are meant to parse. Does anybody know of a resource like this?

thanks,j

2

@treehouse,

We do have some GROK/ELK examples online here:


Are there some standard formats that you would like us to include there?

I understand that you are looking to handle a custom problem here, so let me also answer that question. With GROK match, you can have more than one pattern, so that multiple things will match like this:

filter {
  grok {
    match => { "message" => [
        "%{IPORHOST:clientip} \[%{GREEDYDATA:timestamp}\] %{NUMBER:response:int} %{NUMBER:bytes}",
        "%{IPORHOST:clientip} \[%{GREEDYDATA:timestamp}\] %{NUMBER:response:int} -"
        ]}
  }
}

Then, your date can be parsed using the [logstash date filter] (https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match) like this:

  date {
      match => [ "timestamp" , "E, d MMM y HH:mm:ss z" ]
      locale => "en"
  }

See Also:
http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html

1 Like