Logstash grok match pattern for message field

Elastic Stack Logstash
21

What operating system are you running this on? Are you specifying the number of worker threads when you start Logstash?

22

Windows Operating system. Default thread am using(i.e only 1).

23

@Christian_Dahlqvist, yes i just tested with Logstash 1.5.2, I have the issue for below log document.

Suppose if log document is:

2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85

Loading JSON is:

   {
       "message" => "2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04
e2-f409-ffff08d20b85\r",
      "@version" => "1",
    "@timestamp" => "2015-07-11T12:07:02.733Z",
          "host" => "AnilKumar-PC",
          "path" => "F:/Log Analysis Tool/logs/AKPOS_MessageWS_Log4Net.log.20150415_bak",
          "time" => "2015-01-31 15:58:56,851",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "Setting offline returninvoice:"
}
24

It does look strange that only part of the line seem to be matched by the GREEDYDATA grok expression. I do unfortunately not have any Windows machine available so can not try to replicate your setup.

25

As I said yesterday, you have a trailing space in your grok expression (right after %{GREEDYDATA:msg}). That causes your message to get truncated at the last space unless there are trailing spaces.

26

@magnusbaeck @Christian_Dahlqvist

Please find the below configuration file.

input {

    file {

      path => [ "\\logfilepath.*_bak" ]
      start_position => "beginning"     
     
     }
 }

filter {

multiline{
        pattern => "^%{TIMESTAMP_ISO8601}"
        what => "previous"
        negate=> true
    }

  # for removing the trailing white spaces
  mutate {
    strip => "message"
  }

mutate {
    gsub => ['message', "\n", " "]
}

mutate {
    gsub => ['message', "\r", " "]
}

grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg} " } 
  }
}

  output {
    
    	elasticsearch {
                bind_host => "127.0.0.1"
                port => "9200"
                protocol => http
           }
    	stdout { codec => rubydebug }
        }

Some data is trimmed in msg filed as i said earlier...

27

For the third time, you have a trailing space in your grok expression (right after %{GREEDYDATA:msg}). Remove it.

28

Yes @magnusbaeck, I already tested with that also. I am getting same problem with that also. Updated pattern is find below

   grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
  }
29

Please give example input that exhibits the problem. I can't reproduce with what you've provided earlier (the "Setting offline returninvoice" line).

30

See the below output.

{
           "message" => "2015-04-15 04:48:43,882 [8] DEBUG NCR.XE.Component.MessageHandler.HistoryEventMessageProcessor - Processing 0 promo lines for Invoice_IID c0000000-417a-1638-e905-ffff08d24556",
          "@version" => "1",
        "@timestamp" => "2015-07-14T06:17:36.472Z",
              "host" => "MSSPC036",
              "path" => "",
              "time" => "2015-04-15 04:48:43,882",
            "thread" => "8",
          "loglevel" => "DEBUG",
             "class" => "NCR.XE.Component.MessageHandler.HistoryEventMessageProcessor",
               "msg" => "Processing 0 promo lines for Invoice_IID"
    }
31

Works fine for me. I still think you're using a grok expression with a trailing space. To be extremely clear, this is what your grok filter must look like (and it's what I've used successfully).

grok { 
  match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
}
32

Thanks @magnusbaeck , Now it's working fine....I just take the grok filter from your post it's working now. Great work magnus thank you...

33

Hi all,
@abathula
I am also stuck in finding pattern for my logs.
Please help.

Logs are of type:

"[INFO ] [19:37:08.327] [RSP] |ADD|IOrd 8|EOrd 1100000000011338|ESeq 76038132743205|Strm 3|Error 0|ProcessExchAddResp"

I have made grok pattern like this:

grok { match => { "message" => " %{SYSLOG5424SD:syslog1} %{SYSLOG5424SD:syslog2} %{SYSLOG5424SD:syslog3}%{LOGLEVEL:loglevel} %{GREEDYDATA:junk1} IOrd %{INT:order} %{GREEDYDATA:junk2}" ]"}}

Basically, I want a new field for IOrd whose value is 8 in this log, so that I can search in my Kibana specific value of IOrd.

Thanks in advance

34

@amits, please start a new thread for your unrelated question.