So I have a basic http json post that is sent on the format of
{
"sq": 3,
"ht":
[
{ "si": "90:B6:86:42:2C:9F",
"sm": "90B686",
"bi": "00:00:00:00:00:00",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -90,
"sl": -90,
"ss": -90,
"cn": 1
},
{ "si": "60:A3:7D:D8:A7:F0",
"sm": "60A37D",
"bi": "28:28:5D:69:14:0A",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -77,
"sl": -77,
"ss": -77,
"cn": 1
},
{ "si": "60:D9:C7:9E:DC:E4",
"sm": "60D9C7",
"bi": "28:28:5D:69:14:0A",
"ap": 0,
"ot": 1473246736,
"ct": 1473246736,
"sh": -88,
"sl": -88,
"ss": -88,
"cn": 1 }
],
"vs": 3,
"zver": 1,
"pf": 12,
"sn": "84:24:8D:86:45:84"
}
The ht is the container with the JSON in, what I am trying to do is cycle through each set of data inside "ht" and store as its own document within elastic search, each document should also contain the root level "sn" field and a time stamp.
So if i setup logstash to receive the JSON and use a GROK filter with GREEDYDATA it will store the whole thing in a document. for example:
{
"_index": "logstash-pbtest-2019.11.08",
"_type": "_doc",
"_id": "U4QiTW4B9XJ94OOF--q6",
"_version": 1,
"_score": null,
"_source": {
"vs": 3,
"@version": "1",
"ht": [
{
"si": "6C:3B:6B:60:E4:31",
"ss": -91,
"ot": 1573252635,
"sl": -91,
"sh": -91,
"sm": "6C3B6B",
"bi": "6C:3B:6B:60:E4:31",
"ap": 1,
"ct": 1573252635,
"cn": 1
},
{
"si": "6C:3B:6B:60:E3:E6",
"ss": -90,
"ot": 1573252635,
"sl": -90,
"sh": -90,
"sm": "6C3B6B",
"bi": "6C:3B:6B:60:E3:E6",
"ap": 1,
"ct": 1573252635,
"cn": 1
},
{
"si": "AE:86:74:9D:0E:27",
"ss": -30,
"ot": 1573252635,
"sl": -30,
"sh": -30,
"sm": "AE8674",
"bi": "AE:86:74:9D:0E:27",
"ap": 1,
"ct": 1573252635,
"cn": 1
}
],
"pf": 12,
"headers": {
"http_user_agent": null,
"http_version": "HTTP/1.1",
"http_accept": "/",
"content_type": "application/json",
"request_path": "/",
"request_method": "POST",
"http_host": "10.129.49.9:8880",
"content_length": "4748"
},
"zver": 1,
"@timestamp": "2019-11-08T22:29:28.527Z",
"sn": "74:67:F7:03:BC:18",
"host": "10.2.97.5",
"sq": 29
},
"fields": {
"@timestamp": [
"2019-11-08T22:29:28.527Z"
]
},
"sort": [
1573252168527
]
}
When I come to break this down i use a basic filter / logstash config:
input {
http {
port => 8880
codec => json
}
}
filter {
json {
source => "[ht]"
target => "pres"
}
}
output {
elasticsearch {
hosts => ["elastic01.enc.irondrv.com:9200"]
index => ["logstash-pbtest-%{+YYYY.MM.dd}"]
}
stdout {
codec => rubydebug
}
}
This is where i run into the problem..
I keep getting the following example logged.
[2019-11-08T22:42:44,298][WARN ][logstash.filters.json ][main] Error parsing json {:source=>"[ht]", :raw=>[ {"sm"=>"E68D8C", "si"=>"E6:8D:8C:CB:75:54", "sh"=>-83, "sl"=>-83, "ss"=>-83, "ap"=>1, "bi"=>"E6:8D:8C:CB:75:54", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:14", "sh"=>-49, "sl"=>-49, "ss"=>-49, "ap"=>1, "bi"=>"74:67:F7:2A:8E:14", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:13", "sh"=>-52, "sl"=>-52, "ss"=>-52, "ap"=>1, "bi"=>"74:67:F7:2A:8E:13", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:8E:10", "sh"=>-30, "sl"=>-30, "ss"=>-30, "ap"=>1, "bi"=>"74:67:F7:2A:8E:10", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}, {"sm"=>"7467F7", "si"=>"74:67:F7:2A:D7:F0", "sh"=>-10, "sl"=>-10, "ss"=>-10, "ap"=>1, "bi"=>"74:67:F7:2A:D7:F0", "ct"=>1573253431, "ot"=>1573253431, "cn"=>1}], :exception=>java.lang.ClassCastException: class org.jruby.RubyArray cannot be cast to class org.jruby.RubyIO (org.jruby.RubyArray and org.jruby.RubyIO are in unnamed module of loader 'app')}
Which I cannot work out why. Obviously its matching the field containing the JSON, but am I making a rooky mistake?
Any help would be appreciated.
I'm using logstash 7.4.2 on Ubuntu 18.04 installed via APT am I missing a plugin or Ruby app?