I am trying to parse json which is divided by delimiter($). Does someone have idea how I can configure my logstash?
Example:
My log includes:
ReportBody:{"prop1": val1, "prop2":val2}${"prop1": val3, "prop2":val4}${"prop1": val5, "prop2":val6}
I want it to display in Kibana (Discover) by having each "prop1" indexed and "prop2", so that each part of the delimited json will appear separately. Any ideas?
I advice this configuration :
grok {
match => ["message", "ReportBody:%{GREEDYDATA:json}"]
}
split {
field => "json"
terminator => "$"
}
json {
source => "json"
}
mutate {
remove_field => ["message","json"]
}Thanks Fabian. I tried what you mentioned below. It seems to get a little further, but I don't see that each of the parts still display in Discover. I tried to be even clearer below and tell you exactly what I am looking for. Obviously, I need this breakdown, so I can use it for dashboard. If there are any ideas that would be great. Thanks so much!
From my example:
{ "count" : 1 , "prop2" : "val2"}${ "count" : 2 , "prop2" : "val3"}${ "count" : 3, "prop2" : "val4"}${ "count" : 4 , "prop2" : "val5"}
I see in Discover:
"count" 1
"prop2" "val2"
"message" { "count" : 2 , "prop2" : "val3"}
(I don't see the rest of the lines from ReportBody in Discover.
What I was hoping to see something like:
"count"[1] = 1
"prop2"[1]="val2"
"count"[2] = 2
"prop2"[2]="val3"
"count"[3] = 3
"prop2"[3]="val4".
In actuality, what I really want in Discover is:
"val2"=1 <which equals to count[1]>
"val3"=2 <which equals to count[2]>
"val4"=3 <which equals to count[3]>
"val5"=4 <which equals to count[4]>
One more addition - The message (when I don't remove at the end of logstash config) does show in its entirety, not like what I show above. Otherwise, I am having same issues as above. Any help would be great.
The main question : when you have a log line with 2 "$", what do you want in Kibana ?
- 3 documents (one document per Json part)
- only one document which joins the 3 Json parts
What you need is not easy, but I think I managed to do it.
I tried to do it without using ruby plugin, but for your final need, I have no simpler solution.
grok {
match => ["message", "ReportBody:%{GREEDYDATA:json}"]
}
mutate {
gsub => [ "json" , "\$", "," ]
}
mutate {
replace => { "json" => "[%{json}]" }
}
json {
source => "json"
target => "json"
}
ruby {
code => "event['json'].each { |item| event[item['prop2']] = item['count'] }"
}
mutate {
remove_field => ["message","json"]
}Thanks alot. I so appreciate the help. I think it is closer and may be the solution. However, when I bring up logstash now, I get the following error:
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected character ('|' (code 124)): was expecting comma to separate ARRAY entries
I know I sent you the ReportBody, but is part of message, that includes "|" between the parameters
Example of message:
[FlowContextId1] Host:abc1 | Type:Report | Status:Success | ReportName:Count report | ReportBody:{"prop1": val1, "prop2":val2}${"prop1": val3, "prop2":val4}${"prop1": val5, "prop2":val6} | Failures:0
--Sorry didn't include whole message before, just thought ReportBody is important.
OK, so try this :
grok {
match => ["message", "ReportBody:%{DATA:json} \|"]
}
mutate {
gsub => [ "json" , "\$", "," ]
}
mutate {
replace => { "json" => "[%{json}]" }
}
json {
source => "json"
target => "json"
}
ruby {
code => "event['json'].each { |item| event[item['prop2']] = item['count'] }"
}
mutate {
remove_field => ["message","json"]
}I just tried it and restarted logstash. I get:
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for ARRAY (from [Source: [B@64477cb0; line: 1, column: 0])
at [Source: [B@64477cb0; line: 1, column: 185]>, :level=>:warn}
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for ARRAY (from [Source: [B@2cef9469; line: 1, column: 0])
at [Source: [B@2cef9469; line: 1, column: 537]>, :level=>:warn}
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for ARRAY (from [Source: [B@4bd479ed; line: 1, column: 0])
at [Source: [B@4bd479ed; line: 1, column: 69]>, :level=>:warn}
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for ARRAY (from [Source: [B@5bc85833; line: 1, column: 0])
at [Source: [B@5bc85833; line: 1, column: 185]>, :level=>:warn}
Any ideas? Thanks.
It's really surprising to see raw=>{} because it means json is empty.
Is it you who remove content for confidential reasons ?
If it is the reason, could you give the content, replacing confidential data by "value1", ... ?
Another surprising thing : raw content should be like [{json1},{json2}]
Upon your log, it seems there are no brackets around json.
If your log line is equals to this :
[FlowContextId1] Host:abc1 | Type:Report | Status:Success | ReportName:Count report | ReportBody:{"prop1": val1, "prop2":val2}${"prop1": val3, "prop2":val4}${"prop1": val5, "prop2":val6} | Failures:0
The problem is that you json is invalid, because val1, val2, ... are not quoted strings.
In your real data, is it a number or a real string ?
If it is a real string, it must be quoted to be valid.
The good format is :
[FlowContextId1] Host:abc1 | Type:Report | Status:Success | ReportName:Count report | ReportBody:{"prop1": "val1", "prop2":"val2"}${"prop1": "val3", "prop2":"val4"}${"prop1": "val5", "prop2":"val6"} | Failures:0
I try with this line and it works.
To be clearer "prop1" is really "count" and its value is an integer. However, the value of "prop2" is a string. When I just included the first line of the configuration that you sent, so I can see what json I receive, I got:
json:
{ "count" : 1 , "prop2" : "value1"}${ "count" : 9 , "prop2" : "value2"}${ "count" : 11 , "prop2" : "value3"}${ "count" : 15 , "prop2" : "value4"}. Is there something I need to change in the logstash config that you gave me, since my json values include integers and strings?
Thanks.
There is nothing to do in logstash to process numbers and strings, this works fine since number values are not quoted and string values are quoted.
That said, If you still have json parse errors and want I help you, you'll have to give more details than :
What is the real "raw" referenced in json filter error ? It is not just {} ?
What is the exact initial log line in input ?
The json that displays in Discover is:
[{ "count" : 1 , "deviceType" : "iPad.3.0"},{ "count" : 9 , "deviceType" : "Android.6.0"},{ "count" : 9 ,
"deviceType" : "iPad.3.1"},{ "count" : 9 , "deviceType" : "Android.7.0"}
I see I am not getting end of array. Therefore, if I then comment back in the following ( before I commented out, because I wanted to see what is the json I get), I get the error below, "json {
source => "json"
target => "json"
}
The error is (the raw{} is blank because it can't read json, I guess:
Trouble parsing json {:source=>"json", :raw=>{}, :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for ARRAY (from [Source: [B@26b4958b; line: 1, column: 0])
at [Source: [B@26b4958b; line: 1, column: 185]>, :level=>:warn}
I tried different ways in logstash config to display the end of array, but didn't. Any ideas? Your help is great.
Hi,
I tried to put in input this message :
[FlowContextId1] Host:abc1 | Type:Report | Status:Success | ReportName:Count report | ReportBody:{ "count" : 1 , "deviceType" : "iPad.3.0"}${ "count" : 9 , "deviceType" : "Android.6.0"}${ "count" : 9 ,"deviceType" : "iPad.3.1"}${ "count" : 9 , "deviceType" : "Android.7.0"} | Failures:0
If you have still problems, I'm invite you to give me your input log line.
And it works perfectly (with exactly the same configuration I indicated previously).
At the end, I've got this :
"iPad.3.0" => 1,
"Android.6.0" => 9,
"iPad.3.1" => 9,
"Android.7.0" => 9
I invite you to test with this input to see the result (and tell me) :
output {
stdout {
codec => rubydebug { }
}
}
That said, you try to put in elasticsearch fields with . characters in their names.
That's a problem.
So at filter chain end, I invite you to add :
de_dot {
}
This will replace '.' by '_'.
To finish, it's weird to use "iPad.3.0" as a field name...
That doesn't look like a field name, but more like a field value.
Maybe, it's more relevant to make analytics processing in elasticsearch, using kibana.
The missing closing bracket is surprising.
Are you sure you don't forget closing bracket here :
replace =>; { "json" > "[%{json}]" }
Fabien,
I have been trying to figure out why I am getting this error of the missing bracket, but haven't been successful. Below is what I get in standard output that you requested, IF I comment out the section in config regarding source/target, otherwise I get the error that it is looking for the end bracket.
LOG:
"FCID" => "573348D284AEAEF5C842BFED",
"ReportName" => "Count report",
"ReportBody" => "{ "count" : 1 , "deviceType" : "iPad.3_0"}${ "count" : 9 , "deviceType" : "iPad.3_1"}${ "count" : 9 , "deviceType" : "iPad.3_2"}${ "count" : 9 , "deviceType" : "iPad.3-3"}",
"Failures" => "0",
"result" => "ok",
"json" => "[{ "count" : 1 , "deviceType" : "iPad.3_0"}${ "count" : 9 , "deviceType" : "iPad.3_1"}${ "count" : 9 , "deviceType" : "iPad.3_2"}${ "count" : 9 , "deviceType" : "iPad.3-3"}"
Configuration:
grok {
match => ["message", "ReportBody:%{DATA:json} |"]
}
mutate {
gsub => [ "json" , "\$", "," ]
}
mutate {
replace => { "json" => "[%{json}]" }
}
json {
source => "json"
target => "json"
}
ruby {
code => "event['json'].each { |item| event[item['deviceType']] = item['count'] }"
}
mutate {
remove_field => ["message"]
}OK, so we have exactly the same configuration, but it works for me and not for you 
The only reason I see is logstash version.
I use last one (2.3.2).
Which logstash version do you use ?
1.5.3:(I could try to install later version. Thanks so much for all your help!