Iterate parsing json in Logstash

Elastic Stack Logstash
1

Hello All,

My Json field looks like below,
{"group" : [{"name":"group1","tag":[{"name":"tag1"},{"name":"tag2"},{"name":"tag3"},{"name":"tag4"}]}],"association" : }

And my config file having mutate filter is as follows,
mutate{
add_field => {
"group_name" => "%{[group][name]}"
"tag1" => "%{[group][tag][0][name]}"
"tag2" => "%{[group][tag][1][name]}"
"tag3" => "%{[group][tag][2][name]}"
"tag4" => "%{[group][tag][4][name]}"
}
}
This code works fine . However, there are a varying number of values in the "name" field. Ie, some json doesn't have 4 set of name:tagn inside tag and some have additional 5 or 6 set of name:tagn inside tag field. However, for each name, there will definitely be a corresponding tagn in the "value" field.

So this makes me difficult to automate ingestion and everytime I have to make changes in config file.
I heard we can solve it using ruby code. But I am a starter and I do not know much about ruby.

Can anyone help me iterating this based on number of sets of name:tagn available inside tag array using ruby.

2

You could try

    ruby {
        code => '
            a = event.get("[group][0][tag]")
            a.each_index { |x|
                event.set("tag#{x+1}", a[x]["name"])
            }
        '
    }
3

HI,
Thanks for the reply.
I think this will help me iterating. When I am iterating I have to ingest these values to a particular meta.
Currently I tried something like below,
mutate{
add_field => {
"group_name" => "%{[group][name]}"
"test" => "%{[group][tag]}"
"tags" => ""
"tag" => "%{[group][tag][0][name]}"
}
}
ruby {
code => '
a = event.get("[group][0][tag]")
a.each_index { |x|
event.set("tag#{x+1}", a["name"])
}
'
}
And my result is
{
"group_name" => "group1",
"test" => "{name=tag1},{name=tag2},{name=tag3},{name=tag4},
"tag" => "tag1",
"tags" => [
[0] "",
[1] "_rubyexception"
]
}

How can I overcome this and put these values to a particular meta.

4

Please edit your post, select your configuration, and click on </> in the tool bar above the edit pane.

Also, what ruby exception does logstash log?

5

Hello @Badger,
Sorry that I could not edit the previous post.
The ruby exception is as below,
ERROR][logstash.filters.ruby ][test] Ruby exception occurred: undefined method each_index' for nil:NilClass

6

That means [group][0][tag] does not exist. You could test for that in the ruby filter

ruby {
    code => '
        a = event.get("[group][0][tag]")
        if a
            a.each_index { |x|
                event.set("tag#{x+1}", a[x]["name"])
            }
        end
    '
}
7

Hello @Badger,
Thanks for that. It really worked. I tried something below and it is getting results in the format of tag1:name1,tag2:name2

mutate{
    			add_field => {
    				"group_name" => "%{[group][name]}"
    			}
    		}
    		ruby {
    			code => '
    				a = event.get("[group][tag]")
    				a.each_index { |x|
    					event.set("tag#{x+1}", a[x]["name"])
    				}
    			'
        }
    	}
    	prune {
     		whitelist_names => ["group_name","tag"]
    		}
    	fingerprint {
    			method => "SHA256"
    			source => ["group_name","tag"]
    			target => "[@metadata][generate_id]"
    		}

But since I have given prune and fingerprint filters it is filtering the tag1,tag2... fields. Can I specify something common there like tag* or something which will take all the field starting with tag.

8

The prune filter whitelist_names option takes regexps. So "tag" will match "tag1" and "maytag".

The fingerprint you could do using something like

ruby {
    code => '
        a = event.get("[group][tag]")
        if a
            s = ""
            a.each_index { |x|
                event.set("tag#{x+1}", a[x]["name"])
                s += ":" + a[x]["name"]
            }
            event.set("[@metadata][tags]", s)
    '
}
fingerprint {
    method => "SHA256"
    source => ["group_name","[@metadata][tags]"]
    target => "[@metadata][generate_id]"
}

That assumes the [group][tag] array is ordered. If not you will have to change a.each_index to a.sort.each_index

9

Hello @Badger
I have tried like below as suggested by you

mutate{
add_field => {
"group_name" => "%{[group][name]}"
}
}
ruby {
code => '
a = event.get("[group][tag]")
if a
s = ""
a.each_index { |x|
event.set("tag#{x+1}", a["name"])
s += ":" + a["name"]
}
event.set("[@metadata][tags]", s)
'
}
prune {
whitelist_names => ["group_name","tags"]
}
fingerprint {
method => "SHA256"
source => ["group_name","[@metadata][tags]"]
target => "[@metadata][generate_id]"
}
I am getting the following error in ruby which I could not figure out,
[ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SyntaxError) (ruby filter code):12: syntax error, unexpected end-of-file

Please suggest.

10

Change

        event.set("[@metadata][tags]", s)
'

to

        event.set("[@metadata][tags]", s)
    end
'

That closes out the 'if a' that I added to avoid the

undefined method each_index' for nil:NilClass
11

Hello @Badger,
I tried below adding and

mutate{
add_field => {
"group_name" => "%{[group][name]}"
}
}
ruby {
code => '
a = event.get("[group][tag]")
if a
s = ""
a.each_index { |x|
event.set("tag#{x+1}", a["name"])
s += ":" + a["name"]
}

        event.set("[@metadata][tags]", s)
  	end
  	'
}

prune {
whitelist_names => ["group_name","tags"]
}
fingerprint {
method => "SHA256"
source => ["group_name","[@metadata][tags]"]
target => "[@metadata][generate_id]"
}

But tags field itself is not created and the output is as follows,
{ "group_name" => "name1" }

Kindly suggest.

12

That should be "tag", not "tags". Your field names are tag1, tag2 etc. and they do not start with "tags".

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.