How to parse a json message

Elastic Stack Logstash
1

I have the following input in grok.

{
"payload": {
"user_id": "323232323323223233",
"order_id": "4232313-32323232-232322323",
"ts": "2017-09-01T00:59:00.0138639Z",
"promo_codes": [{
"amount": 10.0,
}],
"shipping_contact_name": "Person X",
}
}

I want the output as mapping for each of the fields. I have used the filters below, but they are not working.

filter {
grok {
match => { "message" => "%{WORD:MyWord} data=%{GREEDYDATA:payload}"} }
json{
source => "request" target => "payload" remove_field => ["request"] }
mutate {
add_field => { "user_id" => "%{[payload][user_id]}" "order_id" => "%{[payload][order_id]}" "ts" => "%{[payload][ts]}" }
}

Can someone help? Thanks

2

This doesn't add up. The grok filter writes to the payload field and the json filter parses the request field and also writes to the payload field.

3

@magnusbaeck
Is the syntax correct for the json though?
Can you access fields in this fashion after parsing to json (if in this case payload were the actual target)?

And to add to this, if the json structure is always the same, but some field names might change, can you do it with an index search?
For instance, in his example, if "order_id" were a dynamic name, but always at the same place, how could you access this field?

4

Is the syntax correct for the json though?

Yes.

Can you access fields in this fashion after parsing to json (if in this case payload were the actual target)?

Yes.

For instance, in his example, if "order_id" were a dynamic name, but always at the same place, how could you access this field?

What does "at the same place" mean? Fields aren't positional so there's no way of saying "grab the value from the second field". If the name is dynamic, how would you know which field you're looking for?

5

What does "at the same place" mean? Fields aren't positional so there's no way of saying "grab the value from the second field". If the name is dynamic, how would you know which field you're looking for?

I see what you mean, I think I'll need some string manipulation for that...
Btw, do you access the fields like this in an "IF" statement?

=> if [jsonData][response][isSuccessful] == true
or
=> if %{[jsonData][response][isSuccessful]} == true

6

The former. See https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html.

7

Thanks.

Sorry to insist, but consider this example:

"response": {
"isSuccessful": true,
"ThisFieldIsDynamic": {
"Header": {
"ID": "something"
}
}
}

Is there a way to access "Header" without knowing the name of "ThisFieldIsDynamic"?
I cannot find examples that work... and a clean json (using ruby?) solution would be awesome!

Something like [response]["regex?"][Header]

8

You'll have to write a piece of Ruby in a ruby filter that iterates over the values of the response hash and checks what type each value is. If it's a hash then look for the Header subfield in that hash.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.