I really struggled with understanding how to work with the JSON data. I first read about the Logstash JSON filter. The doc refers to source => "message" Is this the parent object of the JSON string? What do I do next to get to a point where I have data that is parsed and available for queries? I felt like I was missing the next step and still fail to grasp how to use the JSON Logstash filter. I searched some more and found where a discuss poster had used split on the top level field. I tried that and eventually got to where I could access the fields. I don't know if this is right, best practice or I got really lucky.
I ended up working with this in two different ways.
1: I split on the top level field and then rename like this-
split {field => "data"}
mutate {
rename => { "[data][dsthost]" => "dst_host" }
rename => { "[data][dstport]" => "dstport" }
snip....
Once the fields are renamed I can work with the new fields for data type conversion etc.
The other way I approached this was to still use split but rather that access the data in [parent_tag][child_data_tag] format I did this-
split { field => ["labels"] }
mutate {
convert => {"labels.created_on" => "integer"}
convert => {"labels.last_valid_on" => "integer"}
This approach seemed to work and I didn't have to define tons of rename lines. In the index the fields were dotted labels.name.
I guess my point is that the documentation is really good but I ran into a situation where I did not have a clue what do do next and some concrete examples would have been helpful. If you would like I can help write up a How-To on parse JSON into useful fields for visualization or query so the next person doesn't have to struggle. Please bear in mind that I have been using the ES for about 4 weeks, have not done any training and am solely relying in the Discuss forum and documentation to be my guide. If my efforts are not to far off base with respect to parsing JSON data I would be more than happy to write up a How-To parse and Index JSON and send it to you.