How to grok using json filter

Elastic Stack Logstash
1

Hi,
I have json text (errorCode, errorMessage, etc...) in json text ("raw") and I need to grok this text.
I assume I need to use the json filter for that.
Any chance to get help how to do it?

{
  "timestamp" : 1500659793207,
  "severity" : "Error",
  "userAgent" : "Mozilla/5.0 (Linux; Android 6.0.1; Alcatel_4060O Build/MMB29M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.89 Mobile Safari/537.36",
  "route" : "confirmorder",
  "raw" : {
    "response" : "{\"errorCode\":\"000302\",\"errorMessage\":\"We're sorry, we are currently experiencing some difficulty processing your payment. Please verify your credit card details.\",\"transactionId\":\"eac4c651-c2fa-4ec0-8936-2b6d1cf54adf\"}"
  }

Thanks
Sharon.

1 Like
2

You can use the JSON filter for it, and define your field as the source. E.g.

filter {
  json {
    source => "[raw][responce]"
  }
}
3

Actually this filter isn't working for me.
my problem is even wider.
The logs entry looks like that:
TransactionID: d82fd7c5-41b3-4506-a9f4-5e75f7ed8c60 <FRAMEWORK> <CLIENT_LOG_MSG> ********************************************************************************************************************************* MAC id:ec:9b:f3:d4:a4:1d :: Location:WODTrial :: Geo Code:fc:91:14:8a:3c:63 :: Device model:Samsung Galaxy S6 Edge :: Device Nickname:null :: User Id:TAYLORDUKE2616 :: User-Agent:Mozilla/5.0 (Linux; Android 7.0; SM-G925T Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36 :: Refferer(URI):https://wifiondemand.xfinity.com/wod/ :: Client IP:73.58.158.246, 10.10.10.164, 10.108.2.44 :: Timestamp:2017-07-24 09:43:00.178 ********************************************************************************************************************************* { "timestamp" : 1500907380064, "severity" : "Error", "userAgent" : "Mozilla/5.0 (Linux; Android 7.0; SM-G925T Build/NRD90M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/59.0.3071.125 Mobile Safari/537.36", "route" : "confirmorder", "raw" : { "response" : "{\"errorCode\":\"000302\",\"errorMessage\":\"We're sorry, we are currently experiencing some difficulty processing your payment. Please verify your credit card details.\",\"transactionId\":\"835686ea-b71f 4d04-a339-a91588f8d2e1\"}" }, "template" : "confirm.tmpl" }

I created a grok for that and used it in the grok debugger:

TransactionID: %{DATA:transactionid} \<%{DATA}\> \<%{DATA}\> \*+ MAC id:%{MAC:macid} :: Location:%{LOCALSTRING:location} :: Geo Code:%{MAC:geozone} :: Device model:%{DATA:devicemodel} :: Device Nickname:%{DATA:devicename} :: User Id:%{DATA:userid} :: User-Agent:%{DATA:useragent} :: Refferer\(URI\):%{DATA:reffereruri} :: Client IP:%{DATA:clientip} :: Timestamp:%{TIMESTAMP_ISO8601:timestamp} \*+ \{   "timestamp" : %{BASE10NUM:jepochtimestamp},   "severity" : \"%{LOGLEVEL:jseverity}\",   "userAgent" : \"%{GREEDYDATA:juseragent}\",   "route" : \"%{DATA:jroute}\",   "raw" : \{ %{GREEDYDATA:raw} \},   "template" : \"%{DATA:template}\" \}

Its is working fine and I can see in the below , all the mapping.
The problem is that when I set this grok in the Logstash it is failing to start.
I am getting the following in the Logstash log:

###############  START APPLICATION : OMNI APPSERVER CLIENT #############\n        if [type] == \"omniappservclient\" {\n                mutate {\n                        uppercase => [ \"severity\" ]\n                }\n                grok {\n                      tag_on_failure => [ \"_grokparsefailure\" , \"_urldecodefailure\" , \"_jsonparsefailure\" ]\n                      break_on_match => true\n                      keep_empty_captures => false\n                      match => {\n                           message => [\n                           \"TransactionID: %{DATA:transactionid} \\<%{DATA}\\> \\<%{DATA}\\> \\*+ MAC id:%{MAC:macid} :: Location:%{LOCALSTRING:location} :: Geo Code:%{MAC:geozone} :: Device model:%{DATA:devicemodel} :: Device Nickname:%{DATA:devicename} :: User Id:%{DATA:userid} :: User-Agent:%{DATA:useragent} :: Refferer\\(URI\\):%{DATA:reffereruri} :: Client IP:%{DATA:clientip} :: Timestamp:%{TIMESTAMP_ISO8601:timestamp} \\*+ \\{   \"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:50:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:145:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:286:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:95:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:274:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:185:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

When I remove the last part in the grok, and left it like that:

message => [
                           "TransactionID: %{DATA:transactionid} \<%{DATA}\> \<%{DATA}\> \*+ MAC id:%{MAC:macid} :: Location:%{LOCALSTRING:location} :: Geo Code:%{MAC:geozone} :: Device model:%{DATA:devicemodel} :: Device Nickname:%{DATA:devicename} :: User Id:%{DATA:userid} :: User-Agent:%{DATA:useragent} :: Refferer\(URI\):%{DATA:reffereruri} :: Client IP:%{DATA:clientip} :: Timestamp:%{TIMESTAMP_ISO8601:timestamp} \*+  "

Logstash is starting and running.

Also the following using of the filters:

#                urldecode {
#                   field => { "macid" , "geozone" }
#                    all_fields => true
#                }
#                json {
#                        source => "[raw] [responce]"
#                }

It is not working. I am getting failures.

Need urgent help on that, as I already spent more that a day to try and solve it.

Any idea?

Thanks
Sharon.

4

There are some issues with your pattern that I don't know if are due to bad copy/paste or actually present in the config.

First of all, the below is not a valid pattern, there is no LOCALSTRING pattern by default. Have you defined your own pattern?

Location:%{LOCALSTRING:location}

Also, there are multiple spaces in your pattern where there should be only one, thus not matching anything

As for the Logstash error, it is probably a pattern termination error because of double quotes. Escaping them should work. The full pattern below should work fine, just tested on grokdebug.

TransactionID: %{DATA:transactionid} \<%{DATA}\> \<%{DATA}\> \*+ MAC id:%{MAC:macid} :: Location:%{NOTSPACE:location} :: Geo Code:%{MAC:geozone} :: Device model:%{DATA:devicemodel} :: Device Nickname:%{DATA:devicename} :: User Id:%{DATA:userid} :: User-Agent:%{DATA:useragent} :: Refferer\(URI\):%{DATA:reffereruri} :: Client IP:%{DATA:clientip} :: Timestamp:%{TIMESTAMP_ISO8601:timestamp} \*+ \{ \"timestamp\" : %{BASE10NUM:jepochtimestamp}, \"severity\" : \"%{LOGLEVEL:jseverity}\", \"userAgent\" : \"%{GREEDYDATA:juseragent}\", \"route\" : \"%{DATA:jroute}\", \"raw\" : \{ \"%{GREEDYDATA:raw}\" \}, \"template\" : \"%{DATA:template}\" \}

Also, for reference, it might be easier for you to break the original message into smaller parts and then grok each of them separately, could make debugging easier. E.g. those asterisks could be a good separator.

5

Hi,

I tried to break it for small pieces but still can't cross this rock.
I would like that every field of the json will have a unique target line but in the same entry.
Anyway, nothing is working to me.

This is my input:

Timestamp:2017-05-24 09:43:11.733 "requestUrl" : "ecommerce/user/register", "response" : "{"errorCode":"000202","errorMessage":"One or more inputs are invalid","transactionId":"af280fcb-1e5b-4731-a11e-d11b1f286e34"}"

This is my logstash filter

    if [type] == "json1" {
            mutate {
                    uppercase => [ "severity" ]
            }
            grok {
                  tag_on_failure => ["_grokparsefailure" , "_jsonparsefailure" ]
                  break_on_match => true
                  keep_empty_captures => false
                  match => { "message" => "Timestamp:%{TIMESTAMP_ISO8601:timestamp}  %{GREEDYDATA:jsonstring}"
                  }
                  patterns_dir => "/etc/logstash/patterns"
            }
            date {                      
                  match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss aa" , "yyyy-MM-dd HH:mm:ss.SSS" , "ISO8601" ]
            }
            json {
                  source => "jsonstring"
                  target => "doc"
            }
            mutate {
               add_field => {
                  "requestUrl" => "%{[doc][requestUrl]}"
                  "response" => "%{[doc][response]}"
               }
            }
    }

And this is the output:

2017_07_26_20_49_09_Kibana.jpg952×668 128 KB

Really need help here!!!
Thanks
Sharon.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.