Split array fields in logstash

[Aditya_Srivastava]

I have an input where field and value is
"id"=["0", "7", "11", "39", "70"]

I want to split it and make index as key and split value as value. and then print the index where value is 39.

ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == "39"
event['IndexValue'] = i}
"
}

The below code doesnt work.Can anyone help where my code is going wrong

[magnusbaeck (Magnus Bäck)]

Which Logstash version? What's the error message in the log?

[Aditya_Srivastava]

Hi Magnus,

Logstash version is 2.1.3

This is the exception which I am getting

"tags" => [
[0] "_rubyexception"
],

[magnusbaeck (Magnus Bäck)]

Look in your Logstash log file. It contains details about the Ruby problem.

[Aditya_Srivastava]

Hi magnus,

I am not able too see any logstash config issue.
I will try to give more detail about my ques
my input is
message => field id="0" value="01"/>\n field id="3" value="03"/>\n <field
id="5" value="05"/>\n field id="7" value="07"/>\n <field id="9"
value="09"/>\n /isomsg>\n"

I am applying kv filter here

kv {
source => "message"
trim => """
trimkey => """
field_split => " "
value_split => "="
}

The output I receive is
"id" => [
[0] "0",
[1] "3",
[2] "5",
[3] "7",
[4] "9"
],
"value" => [
[0] "01",
[1] "03",
[2] "05",
[3] "07",
[4] "09"
],

Now I want to write a ruby code where I can extract the id and value wrt to index

The ruby code should be in loop and check , if id[index] == "7"
event['thisvalue'] = value[index]

Means first find the index where id is 7(say indexA is 3) and then extract the value at indexA(which should be 07) and store this value as a field.

For this the code i wrote is

ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == "7"
event['IndexValue'] = i}
"
}
This code should at-least give me index where the value for "id" is 7 which is not happening and I get ruby fail exception as a tag.

"tags" => [
[0] "_rubyexception"
],

[magnusbaeck (Magnus Bäck)]

I am not able too see any logstash config issue.

Look harder. There must be something in the log.

[Aditya_Srivastava]

Which log are you exactly referring to??

[magnusbaeck (Magnus Bäck)]

The Logstash log file, typically in /var/log/logstash. If Logstash misbehaves it's the first thing you should check.

[Aditya_Srivastava]

I have take the logstash zip and unzipped it at a location. I am now running my configuration files from that very location.
bin/logstash -f myfile.conf

So there is no /var/log/logstash which I have. For that matter, I do not see any log file where logstash is dumping any kind of detail.

[magnusbaeck (Magnus Bäck)]

If you run Logstash like that it should log to your terminal.

I suspect the configuration you've posted is different from what your running, because I can spot two errors in your Ruby code that would prevent Logstash from even starting.

[Aditya_Srivastava]

You are correct. It doesnt work for above ruby code.
My logstash config file ran with by commenting few lines.

I took reference from below code written by you for some other post

Iteration in Logstash

I have a string data in Logstash that has been parsed as that looks like "id": "0,3,10" "value": "100,200,300" I would like to match the individual "id" to the corresponding "value" with respect to their order in the string. I want the field to look like "id0": "100" "id3": "200" "id10": "300" As of now, I have used the mutate field to separate the array into its individual values. I used the following code: mutate { add_field => {"id%{[id][0]}" => "%{[value][0]}"} add_field => {"id%…

filter {
ruby {
code => '
ids = event["id"].split(",")
values = event["value"].split(",")
if ids.length == values.length
ids.each_index { |i| event["id#{ids[i]}"] = values[i] }
end
'
}
}

Could you please tell me what is wrong in the ruby code I wrote or help me with ruby code to solve the prob

[magnusbaeck (Magnus Bäck)]

There are at least these two problems:

• You can't use double quotes in the Ruby code since you're using double quotes as the Ruby code delimiter, i.e. replace if times[i] == "7" with if times[i] == '7'.
• There's an end missing at the end of the whole expression.

[Aditya_Srivastava]

Thanks for pointing out the mistakes.
ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == '7'
event['IndexValue'] = i}
end
"
}

Now when I run this, I get foll exception

SyntaxError: (ruby filter code):5: syntax error, unexpected tRCURLY
event['IndexValue'] = i}
^
eval at org/jruby/RubyKernel.java:1079
register at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-2.0.3/lib/logstash/filters/ruby.rb:29
each at org/jruby/RubyArray.java:1613
start_filters at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:171
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:101
execute at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/agent.rb:165
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:90
call at org/jruby/RubyProc.java:281
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:95
call at org/jruby/RubyProc.java:281
initialize at /data/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24

I looked up for the exception, found a solution given by you.

Unexpected error in ruby code in logstash

Hi, An unexpected error in occuring for the following ruby code in logstash filter. SyntaxError: (ruby filter code):5: syntax error, unexpected tRCURLY ** }** Following is the ruby code: ruby { code => ' allscores = event["score"] if allscores.length != '1' t = allscores.inject(:+) event["total_score"] = t ' add_tag => [sum_score] } The error is as follows: SyntaxError: (ruby filter code):5: syntax error, unexpected tRCURLY } ^ eval at org/jruby/RubyKernel.java:1…

ruby {
code => "
times = event["id"].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == '7'
event['IndexValue.to_i'] = i}
end
"
}

when change event['id'] from single quote to double quotes as mentioned above, it gives me foll exception

Error: Expected one of #, {, } at line 25, column 16 (byte 545)

If I comment out the complete ruby code, It works fine. Means again there is some prob with the ruby code.

Plz refer some link as well where I can study more detail on how to use ruby in logstash.

[Nico-DF]

@Aditya_Srivastava

ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == '7'
event['IndexValue'] = i}
end
"
}

There's a problem here:

times.each_index { |i| event[i.to_s] = times[i]
  if times[i] == '7'
    event['IndexValue'] = i}
  end

Try:

times.each_index { |i| event[i.to_s] = times[i]
  if times[i] == '7'
    event['IndexValue'] = i
  end
}

Or close your brace before (I don't know what you're trying to do)

[magnusbaeck (Magnus Bäck)]

Replace

if times[i] == '7'
  event['IndexValue'] = i}
end

with

if times[i] == '7'
  event['IndexValue'] = i
end
}

Plz refer some link as well where I can study more detail on how to use ruby in logstash.

Apart from the ruby filter documentation there isn't much available that applies without reservations to Logstash 2.1.

[Aditya_Srivastava]

Yes. The solution works and the config file runs properly
But still receiving

"tags" => [
[0] "_rubyexception"

Also this message gets logged on console in red color

Ruby exception occurred: undefined method split' for nil:NilClass {:level=>:error} Ruby exception occurred: undefined methodsplit' for ["0", "3", "5", "7", "9"]:Array {:level=>:error}

[Aditya_Srivastava]

@Nico-DF

Or close your brace before (I don't know what you're trying to do)

If possible request you to start fresh as this is getting messy. I will try and frame my query in more clear and structured way now

[Aditya_Srivastava]

@Aditya_Srivastava

"id" => [
[0] "0",
[1] "3",
[2] "5",
[3] "7",
[4] "9"
],

Is this in array format? Can I apply split function on this event.
I feel somehow the ruby exception is because of this array that I am trying to split
Please confirm

[magnusbaeck (Magnus Bäck)]

Ruby exception occurred: undefined method split' for nil:NilClass {:level=>:error}

That error indicates that not all your events have an id field. event['id'] evaluates to nil when the field doesn't exist.

Ruby exception occurred: undefined methodsplit' for ["0", "3", "5", "7", "9"]:Array {:level=>:error}

You can split strings but not arrays. Arrays don't need to be split.

[Aditya_Srivastava]

@magnusbaeck

You can split strings but not arrays. Arrays don't need to be split.

What should I do then if I want to find the index of "7" in given array??