# Logstash error using \`split\` on JSON array, where array is converted from hash **URL:** \ **Category:** Logstash\ **Created:** [November 11, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395 "2019-11-11T16:58:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 11, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/1 "2019-11-11T16:58:32Z")
# TL;DR Receiving errors when using logstash `split` filter after using `ruby` filter to convert JSON dict (hash) to array. # The rest of the story... I have Logstash polling an application for mbean metrics. The data set is coming to logstash in the following format: ```json { "request": { "mbean": "mbean_name", "type": "read" }, "value": { "mbean_id1": { "metric1": true, "metric2": 0, "metric3": 0.0 }, "mbean_id2": { "metric1": true, "metric2": 0, "metric3": 0.0 } } } ``` I need to be able to split the values into individual records. The best way to do this appears to be the `split` filter, which, unfortunately, doesn't work on a dict. So what I am attempting to do is convert the dict to an array, then split the subsequent array. My `conf` file is: ```auto input { file { path => "${CONF_D}/sample-data.json" mode => "read" } } filter { # Convert JSON dict/hash into array, then add it as a new field. ruby { code => " val_a = Array.new event['value'].each do |k, v| v['val_id'] = k val_a.push(v) end event['val_a'] = val_a " } split { field => "val_a" } } output { stdout { codec => rubydebug } } ``` When I run this rule set, I get multiple instances of the following errors: ```auto [2019-11-11T16:10:40,666][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `[]' for # [2019-11-11T16:10:40,668][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `[]' for # [2019-11-11T16:10:40,697][WARN][logstash.filters.split] Only String and Array types are splittable. field:val_a is of type = NilClass [2019-11-11T16:10:40,703][WARN][logstash.filters.split] Only String and Array types are splittable. field:val_a is of type = NilClass ``` However, when I run the following Ruby script against the same data set, processes correctly and I get the following output: ```ruby #!/usr/bin/ruby require 'json' file = File.read('sample-data.json') event = JSON.parse(file) vals_a = Array.new event['value'].each do |k, v| v['val_id'] = k vals_a.push(v) end print vals_a ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 11, 2019, 5:21pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/2 "2019-11-11T17:21:18Z")
You cannot access fields on an event using array syntax (that was eliminated a couple of years back). You need to use the event API. Try ``` ruby { code => ' val_a = [] event.get("value").each { |k, v| v["val_id"] = k val_a << v } event.set("val_a", val_a) ' } ``` ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 11, 2019, 5:46pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/3 "2019-11-11T17:46:19Z")
Woo-hoo! Progress! I've updated the filter as you suggested, and now I'm getting these errors instead: ```auto [2019-11-11T17:44:11,513][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass [2019-11-11T17:44:11,515][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass [2019-11-11T17:44:11,527][WARN][logstash.filters.split] Only String and Array types are splittable. field:val_a is of type = NilClass [2019-11-11T17:44:11,533][WARN][logstash.filters.split] Only String and Array types are splittable. field:val_a is of type = NilClass ``` The full conf file now reads: ```auto input { file { path => "${CONF_D}/sample-data.json" mode => "read" } } filter { # Convert JSON dict/hash into array. ruby { code => ' val_a = [] event.get("value").each { |k, v| v["val_id"] = k val_a << v } event.set("val_a", val_a) ' } split { field => "val_a" } } output { stdout { codec => rubydebug } } ``` ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 11, 2019, 6:34pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/4 "2019-11-11T18:34:56Z")
I thought that the reason for this latest round of errors might have been that I need to use the JSON codec for the input, but when I update my input block to the following: ```auto input { file { path => "${CONF_D}/activemq-sample-data.json" mode => "read" codec => "json" } } ``` I then get the following errors instead: ```auto [2019-11-11T18:25:26,482][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#, :data=>" \"MessageGroups\": {},"} [2019-11-11T18:25:26,495][ERROR][logstash.codecs.json] JSON parse error, original data now in message field {:error=>#, :data=>" \"ConsumerCount\": 0,"} ``` ...so that doesn't appear to be it. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 11, 2019, 7:05pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/5 "2019-11-11T19:05:47Z")
You definitely need either a codec or a json filter. It looks like you and the codec disagree about whether the input is valid JSON. My money would be on the codec 🙂 ---
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [November 11, 2019, 7:26pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/6 "2019-11-11T19:26:23Z")
My money is on the codec being right, as well. I actually realized that it was attempting to parse the input file line-by-line. Since I'm going to be posting the data to logstash anyway, I updated my `input` block to the following: ```auto input { http { codec => "json" port => 8888 host => "0.0.0.0" } } ``` Now I'm just posting the data with: ```auto curl -X POST -H "Content-Type: application/json" -d @"$(pwd)/sample-data.json" localhost:8888 ``` It's still not quite parsing the way I want to, but at least I can see the data coming through the pipeline. I went ahead and accepted your earlier answer, because it resolved the initial issue I requested help with. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 9, 2019, 7:26pm UTC](https://discuss.elastic.co/t/logstash-error-using-split-on-json-array-where-array-is-converted-from-hash/207395/7 "2019-12-09T19:26:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.