Parsing json with delimiters using logstash

Elastic Stack Logstash
21

I upgraded my logstash version and now it works! Thanks! One more question...If I now have device@ in my discover for that request, but as I plan to have fields with same device, but different version also, then I want to show it in pie chart in Kibana. What suggestion do you have of how I can show this in Kibana? For example lets say, I have Android@3_2 and it has count of 4, then I have Android@4_5 and it has count of 6. Now, I want to show them both in one pie chart, with split between the two. I do not know from the beginning which versions will be coming up, I just want all Android devices, in one pie chart (so I could have split of 2 or 10 - depending on how many versions for that same type of device). Any suggestions? Thanks.

22

Nice to see that last logstash version solve your problem.

Concerning your question, to be honest, up to me, you're not storing documents in elasticsearch using the right way so that you can use them efficiently in Kibana.

To me, the right configuration for efficient use in Kibana is the following (I firstly adviced to you) :

grok {
  match => ["message", "ReportBody:%{DATA:json} \|"]
}

split {
  field => "json"
  terminator => "$"
}
  
json {
  source => "json"
}

mutate {
  remove_field => ["message","json"]
}

Using that, for this log line ({ "count" : 1 , "deviceType" : "iPad.3_0"}${ "count" : 9 , "deviceType" : "iPad.3_1"}${ "count" : 9 , "deviceType" : "iPad.3_2"}${ "count" : 9 , "deviceType" : "iPad.3-3"}), you will have these 4 distinct documents in elasticsearch :

{ "count" : 1 , "deviceType" : "iPad.3_0"}
{ "count" : 9 , "deviceType" : "iPad.3_1"}
{ "count" : 9 , "deviceType" : "iPad.3_2"}
{ "count" : 9 , "deviceType" : "iPad.3-3"}

With these 4 documents, you can use all the power of Kibana aggregations :

  • filter on "deviceType:iPad"
  • use term aggregation on deviceType
  • use sum aggregation on count
  • and display nice pies using the result data.
23

This is terrific. Now, I can get graphs according to device types. However, I have numerous deviceTypes, with various versions (such as Android@3.5, Android@4.5, iOS@6.1, iOS@6.2). Currently, each of the deviceTypes and versions display in their own vertical bar. Therefore, now I have 4 vertical bars, where each displays the count. However, I want to be able to see them in a vertical bar graph, where the androids are all counted in one vertical bar (sub-counted by each of its versions - Android@3.5 - count 4 and Android@4.5 - count:2) and then iOS in its own bar, also subdivided by its versions). Then, only 2 vertical bars display for these figures. Can you give me an idea, how I can set my visualization up for this? Thanks.

24

In logstash, I advice you to separate deviceType (ex: Android@3.5) in 2 fields : os (ex: Android) and os_version (ex: 3.5). And of course, you can keep deviceType field.

Once you done that, in Kibana, you can make a first aggregation based on os, then a second on os_version and finally a third on count field.