How to get variable names in nested JSON format

[its-ogawa]

I am using a JSON plugin to parse logs in JSON format.
Eventually, the nested variable names are retrieved along with their parent names.
How can I simply get just the variable names?

target log:
... snip ... {... snip ...,"results":{ ... snip ..., "api_code":"51200"}} ... snip ...

my grok:

grok{
	"match" => { "message" => " ... snip ... (?<MY_JSON>\{.*\})( %{GREEDYDATA:message})? ... snip ..." }
}
if( [MY_JSON] ) {
  json {
    source => "MY_JSON"
  }
}

result:
results.api_code : 51200

expect:
api_code : 51200

[Cad]

Hi,

I think you have three possibilities.

The first one is to update your grok pattern like this :
(?<MY_JSON>\{.*"api_code":"(?<api_code>[0-9]+).*\})
With this syntax, the field MY_JSON don't change and you create a new field named api_code who contain the integer.

The second possibility is to use the rename option with the mutate filter but you have to know in advance the name of each field you need.

mutate {
    rename =>[
        "[MY_JSON][results][api_core]", "api_core"
    ]
}

The last one is to use the ruby filter to browse the JSON.
event['MY_JSON']['results'].each...
With this one, no necessity to know the field name in advance.

I only try the grok pattern so the others possibilities possibly need some update to work.

Cad.

[its-ogawa]

Thanks for answering my question.

Your idea is great!

I tried the grok you tried with Kibana's Grok Debugger and it worked.
However, I actually tried it in logstash and it didn't work for some reason.

my grok:

grok{
  "match" => { "message" => " ... snip ... (?<MY_JSON>\{.*\})( %{GREEDYDATA:message})? ... snip ..."
}
if( [MY_JSON] ) {
  #grok {
  #  "match" => { "MY_JSON" => ".*"api_code":"(?<api_code>[0-9a-zA-Z]+).*" } # <- not working.
  #}
  json {
    source => "MY_JSON"
  }
  mutate {
    add_field => { "api_code" => "%{[results][api_code]}" }
  }
}

Currently, I'm getting the expected results by writing in mutate, but I'm having trouble getting grok to work.

Any advice would be appreciated.
The comment above is currently the best answer.

[Cad]

Hi,

The second pattern not working in the logsatsh file because you need to escape quotes with backslash, my bad.

"match" => { "MY_JSON" => ".*\"api_code\":\"(?<api_code>[0-9a-zA-Z]+)\".*" }

Cad

[its-ogawa]

You're right!

I never realized how simple this is.
Thanks for your sound advice.