Logstash error using `split` on JSON array, where array is converted from hash

Elastic Stack Logstash
1

TL;DR

Receiving errors when using logstash split filter after using ruby filter to convert JSON dict (hash) to array.

The rest of the story...

I have Logstash polling an application for mbean metrics. The data set is coming to logstash in the following format:

{
    "request": {
        "mbean": "mbean_name",
        "type": "read"
    },
    "value": {
        "mbean_id1": {
            "metric1": true,
            "metric2": 0,
            "metric3": 0.0
        },
        "mbean_id2": {
            "metric1": true,
            "metric2": 0,
            "metric3": 0.0
        }
    }
}

I need to be able to split the values into individual records. The best way to do this appears to be the split filter, which, unfortunately, doesn't work on a dict. So what I am attempting to do is convert the dict to an array, then split the subsequent array. My conf file is:

input {
    file {
        path => "${CONF_D}/sample-data.json"
        mode => "read"
    }
}

filter {
    # Convert JSON dict/hash into array, then add it as a new field.
    ruby {
        code => "
            val_a = Array.new
            event['value'].each do |k, v|
                v['val_id'] = k
                val_a.push(v)
            end
            event['val_a'] = val_a
        "
    }

    split {
        field => "val_a"
    }
}

output {
    stdout { codec => rubydebug }
}

When I run this rule set, I get multiple instances of the following errors:

[2019-11-11T16:10:40,666][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `[]' for #<LogStash::Event:0x18b864f7>
[2019-11-11T16:10:40,668][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `[]' for #<LogStash::Event:0x53dc61a9>
[2019-11-11T16:10:40,697][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:val_a is of type = NilClass
[2019-11-11T16:10:40,703][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:val_a is of type = NilClass

However, when I run the following Ruby script against the same data set, processes correctly and I get the following output:

#!/usr/bin/ruby

require 'json'
file = File.read('sample-data.json')
event = JSON.parse(file)

vals_a = Array.new
event['value'].each do |k, v|
    v['val_id'] = k
    vals_a.push(v)
end

print vals_a
2

You cannot access fields on an event using array syntax (that was eliminated a couple of years back). You need to use the event API. Try

    ruby {
        code => '
            val_a = []
            event.get("value").each { |k, v|
                v["val_id"] = k
                val_a << v
            }
            event.set("val_a", val_a)
        '
    }
3

Woo-hoo! Progress!

I've updated the filter as you suggested, and now I'm getting these errors instead:

[2019-11-11T17:44:11,513][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `each' for nil:NilClass
[2019-11-11T17:44:11,515][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `each' for nil:NilClass
[2019-11-11T17:44:11,527][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:val_a is of type = NilClass
[2019-11-11T17:44:11,533][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:val_a is of type = NilClass

The full conf file now reads:

input {
    file {
        path => "${CONF_D}/sample-data.json"
        mode => "read"
    }
}

filter {
    # Convert JSON dict/hash into array.
    ruby {
        code => '
            val_a = []
            event.get("value").each { |k, v|
                v["val_id"] = k
                val_a << v
            }
            event.set("val_a", val_a)
        '
    }
    
    split {
        field => "val_a"
    }
}

output {
    stdout { codec => rubydebug }
}
4

I thought that the reason for this latest round of errors might have been that I need to use the JSON codec for the input, but when I update my input block to the following:

input {
    file {
        path => "${CONF_D}/activemq-sample-data.json"
        mode => "read"
        codec => "json"
    }
}

I then get the following errors instead:

[2019-11-11T18:25:26,482][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"      \"MessageGroups\": {},"}
[2019-11-11T18:25:26,495][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>"      \"ConsumerCount\": 0,"}

...so that doesn't appear to be it.

5

You definitely need either a codec or a json filter. It looks like you and the codec disagree about whether the input is valid JSON. My money would be on the codec :slight_smile:

6

My money is on the codec being right, as well.

I actually realized that it was attempting to parse the input file line-by-line. Since I'm going to be posting the data to logstash anyway, I updated my input block to the following:

input {
    http {
        codec => "json"
        port => 8888
        host => "0.0.0.0"
    }
}

Now I'm just posting the data with:

curl -X POST -H "Content-Type: application/json" -d @"$(pwd)/sample-data.json" localhost:8888

It's still not quite parsing the way I want to, but at least I can see the data coming through the pipeline.

I went ahead and accepted your earlier answer, because it resolved the initial issue I requested help with.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.