Grok pattern is not working

Elastic Stack Logstash
1

Hi All,

I need help on grok patterns. I have a log file where few fields are missing(several fields actually), based on that I have come up with below grok patterns but its not working. In the grok contructor - it showing as matched but most fields are showing blank.

for example AlgoString in first line & Client are showing blank. Please anyone help on this.

logs lines:

07:11:02.002015|INF|RAW(0x2590ed0)|=> Received OrderNew={ Transaction=574F5D76-4BD-3-65 AlgoString=TEST:0:0:50:50 ClOrdId=-1 Client=TEST1
10:30:20.790316|INF|RAW(0x2b77e400ab70)|=> Received OrderNew={ Transaction=574F8C2C-C0E63-7456-65 Price=2121 ClOrdId=1 Client=TEST2
09:11:36.682557|INF|RAW(0x307fbc0)|=> Received OrderNew={ Transaction=574F79B8-A6943-91F-65 Price=664 ClOrdId=32

pattern:

%{TIME:logtime}\|%{DATA:severity}\|.*\|.*Received %{DATA:TransType}={ Tran.*(?: AlgoString=%{DATA:AlgoString}|)?(?: Price=%{NUMBER:Price}|)? Cl.* (?: Client=%{DATA:Client}|)?(?:Account=%{DATA:Account}|)?.*
2

First
I would take your query to https://grokdebug.herokuapp.com/ and test it out

next, there is a "space" before your Client= which is throwing off the query

Next, you are using a lot of greedy statements .* I would replace these with .+ or better yet actual data. \S is nice cause it matches everything not a space. .* has some wierd effects and sometimes matches more then you expect.

Finally, Maybe you don't want to mach the json data with Grok and use the json filter. Something like this would work nicely (oh wait this is not json. Well try the KV filter

*See if something like this helps, Sorry I have some typo's but you should get the jist of what I was doing with it.

filter{
   #get message and create new field with json like data
 grok => ["message", "^%{TIME:logtime}\|%{DATA:severity}\|(|.+)\|.+Received %{DATA:TransType}=%{GREEDYDATA:app_data}"
}
kv{
   source =>"app_data"
}
}

Of course if you do have valid json you can replace kv with the json filter and parsing would be automagic

1 Like
3

Thanks for your reply but I am trying below filter but I donot see any outpu. Any issue withbelow filter condition?

      grok {
              match => [ "message", "^%{TIME:logtime}\|%{DATA:severity}\|(|.+)\|.+Received %{DATA:TransType}=%{GREEDYDATA:app_data}" ]
              add_field => [ "received_at" , "%{@timestamp}" ]
              add_field => [ "received_from" , "%{host}" ]
              add_field => [ "application" , "SOR" ]
      }
	  
      kv {
	       source => "app_data"
	  }
      mutate {
              gsub => ["logtime","\d\d\d$",""]
      }
      date {
            locale => "en"

            match => [ "logtime" , "HH:mm:ss.SSS" ]
            target => "@timestamp"
            timezone => "Asia/Kolkata"
            add_field => { "debug" => "timestampMatched"}
      }
4

Well the regex is working (using the first input of your first post

I put in at Grokdebugger and got this as a result.

{
  "logtime": [
    [
      "07:11:02.002015"
    ]
  ],
  "HOUR": [
    [
      "07"
    ]
  ],
  "MINUTE": [
    [
      "11"
    ]
  ],
  "SECOND": [
    [
      "02.002015"
    ]
  ],
  "severity": [
    [
      "INF"
    ]
  ],
  "TransType": [
    [
      "OrderNew"
    ]
  ],
  "app_data": [
    [
      "{ Transaction=574F5D76-4BD-3-65 AlgoString=TEST:0:0:50:50 ClOrdId=-1 Client=TEST1"
    ]
  ]
}

Can you provide what output you are receiving

output{ stdout{  codec=>rubydebug}}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.