Struggling to parse JSON data with logstash

So I have a basic http json post that is sent on the format of

{
"sq": 3,
"ht":
[
{ "si": "90:B6:86:42:2C:9F",
"sm": "90B686",
"bi": "00:00:00:00:00:00",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -90,
"sl": -90,
"ss": -90,
"cn": 1
},
{ "si": "60:A3:7D:D8:A7:F0",
"sm": "60A37D",
"bi": "28:28:5D:69:14:0A",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -77,
"sl": -77,
"ss": -77,
"cn": 1
},
{ "si": "60:D9:C7:9E:DC:E4",
"sm": "60D9C7",
"bi": "28:28:5D:69:14:0A",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -88,
"sl": -88,
"ss": -88,
"cn": 1 }
],
"vs": 3,
"zver": 1,
"pf": 12,
"sn": "84:24:8D:86:45:84"
}

The ht is the container with the JSON in, what I am trying to do is cycle through each set of data inside "ht" and store as its own document within elastic search, each document should also contain the root level "sn" field and a time stamp.

So if i setup logstash to receive the JSON and use a GROK filter with GREEDYDATA it will store the whole thing in a document. for example:

{
"_index": "logstash-pbtest-2019.11.08",
"_type": "_doc",
"_id": "U4QiTW4B9XJ94OOF--q6",
"_version": 1,
"_score": null,
"_source": {
"vs": 3,
"@version": "1",
"ht": [
{
"si": "6C:3B:6B:60:E4:31",
"ss": -91,
"ot": 1573252635,
"sl": -91,
"sh": -91,
"sm": "6C3B6B",
"bi": "6C:3B:6B:60:E4:31",
"ap": 1,
"ct": 1573252635,
"cn": 1
},
{
"si": "6C:3B:6B:60:E3:E6",
"ss": -90,
"ot": 1573252635,
"sl": -90,
"sh": -90,
"sm": "6C3B6B",
"bi": "6C:3B:6B:60:E3:E6",
"ap": 1,
"ct": 1573252635,
"cn": 1
},
{
"si": "AE:86:74:9D:0E:27",
"ss": -30,
"ot": 1573252635,
"sl": -30,
"sh": -30,
"sm": "AE8674",
"bi": "AE:86:74:9D:0E:27",
"ap": 1,
"ct": 1573252635,
"cn": 1
}
],
"pf": 12,
"headers": {
"http_user_agent": null,
"http_version": "HTTP/1.1",
"http_accept": "/",
"content_type": "application/json",
"request_path": "/",
"request_method": "POST",
"http_host": "10.129.49.9:8880",
"content_length": "4748"
},
"zver": 1,
"@timestamp": "2019-11-08T22:29:28.527Z",
"sn": "74:67:F7:03:BC:18",
"host": "10.2.97.5",
"sq": 29
},
"fields": {
"@timestamp": [
"2019-11-08T22:29:28.527Z"
]
},
"sort": [
1573252168527
]
}

When I come to break this down i use a basic filter / logstash config:

input {
http {
port => 8880
codec => json
}
}
filter {
json {
source => "[ht]"
target => "pres"
}
}
output {
elasticsearch {
hosts => ["elastic01.enc.irondrv.com:9200"]
index => ["logstash-pbtest-%{+YYYY.MM.dd}"]
}
stdout {
codec => rubydebug
}
}

This is where i run into the problem..

I keep getting the following example logged.

[2019-11-08T22:42:44,298][WARN ][logstash.filters.json ][main] Error parsing json {:source=>"[ht]", :raw=>[ {"sm"=>"E68D8C", "si"=>"E6:8D:8C:CB:75:54", "sh"=>-83, "sl"=>-83, "ss"=>-83, "ap"=>1, "bi"=>"E6:8D:8C:CB:75:54", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:14", "sh"=>-49, "sl"=>-49, "ss"=>-49, "ap"=>1, "bi"=>"74:67:F7:2A:8E:14", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:13", "sh"=>-52, "sl"=>-52, "ss"=>-52, "ap"=>1, "bi"=>"74:67:F7:2A:8E:13", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:10", "sh"=>-30, "sl"=>-30, "ss"=>-30, "ap"=>1, "bi"=>"74:67:F7:2A:8E:10", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:D7:F0", "sh"=>-10, "sl"=>-10, "ss"=>-10, "ap"=>1, "bi"=>"74:67:F7:2A:D7:F0", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}], :exception=>java.lang.ClassCastException: class org.jruby.RubyArray cannot be cast to class org.jruby.RubyIO (org.jruby.RubyArray and org.jruby.RubyIO are in unnamed module of loader 'app')}

Which I cannot work out why. Obviously its matching the field containing the JSON, but am I making a rooky mistake?

Any help would be appreciated.

I'm using logstash 7.4.2 on Ubuntu 18.04 installed via APT am I missing a plugin or Ruby app?

Ok my bad.. Needed a split filter..

If anybody has a similar issue.

This is my initial config that works.

input {
http {
port => 8880
codec => json
}

}

filter {
json {
source => "message"
}
split {
field => "ht"
}

}

output {
elasticsearch {
hosts => ["elastic01.enc.irondrv.com:9200"]
index => ["logstash-pbtest-%{+YYYY.MM.dd}"]
}
stdout {
codec => rubydebug
}
}

No, it is not JSON. The "class org.jruby.RubyArray cannot be cast to class org.jruby.RubyIO" is telling you it is an array. That's because it was already parsed by the json codec. You very rarely need both a json codec and a json filter. Remove the json filter.

Cheers for the reply, but the split filter worked for me to resolve, so I assume you are saying the json filter is not actually required?

Correct.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.