Convert netsted array into fields using Logstash

Elastic Stack Logstash
1

Hi,

I'm trying to convert an array with data keys and values into fields with the same name, but keep getting errors or problem with the Ruby filter plugin.

I'll explain with example, I'm trying to take this json:
{
"event_type":"upload",
"traits":[
["status",1,"finished"],
["user_id",1,"1"],
["name",1,"my_image"],
["service",1,"image.localhost"],
["resource_id",1,"ff"],
["created_at",1,"2017-06-29T12:18:48.000000"],
["project_id",1,"1e"],
["size",1,"60817408"]
],
... more fields...
}

Into this:
{
"event_type":"upload",
"status":"finished",
"user_id":"1",
"name":"my_image",
"service":"image.localhost",
"resource_id":"ff",
"created_at":"2017-06-29T12:18:48.000000",
"project_id":"1e",
"size":"60817408",

... more fields...
}

I've added some metadata to relevant messages and tried the following filter:

filter {
    if "" in [@metadata][topicId] {
	ruby {
		code => "event['traits'].each { |k|
				event['k[0]'] = k[2]
			}"
	}
    }
}

But got this error on the log:
[main]>worker3] ERROR logstash.filters.ruby - Ruby exception occurred: Direct event field references (i.e. event['field']) have been disabled in favor of using event get and set methods (e.g. event.get('field')). Please consult the Logstash 5.0 breaking changes documentation for more details.

So I've tried to do it with event.get & event.set methods, but with no success:

filter {     
    ruby {
        code => "
            event.get('traits').each_index {|i|
                event.set([traits][i][0], [traits][i][2])
        }
    }
}

I got:
ERROR logstash.filters.ruby - Ruby exception occurred: undefined local variable or method `traits' for #LogStash::Filters::Ruby:0x3d525d41

Can someone please try to explain me what I did wrong and how to make it right?
If the task is possible to achieve without ruby code, please tell me, it would be great

Thanks,
J

1 Like
2 
event.set([traits][i][0], [traits][i][2])

One of the reasons this doesn't work is because you're mixing Logstash notation into your Ruby code. Try this:

event.get('traits').each { |v|
  event.set(v[0], v[2])
}
1 Like
3

Wow so simple, yet so working..
Thank you very much!
That's exactly what I was trying to do without success

@magnusbaeck
You are truly MAGNUSficent :slight_smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.