Split array fields in logstash


(Aditya Srivastava) #1

I have an input where field and value is
"id"=["0", "7", "11", "39", "70"]

I want to split it and make index as key and split value as value. and then print the index where value is 39.

ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == "39"
event['IndexValue'] = i}
"
}

The below code doesnt work.Can anyone help where my code is going wrong


(Magnus Bäck) #2

Which Logstash version? What's the error message in the log?


(Aditya Srivastava) #3

Hi Magnus,

Logstash version is 2.1.3

This is the exception which I am getting

"tags" => [
[0] "_rubyexception"
],


(Magnus Bäck) #4

Look in your Logstash log file. It contains details about the Ruby problem.


(Aditya Srivastava) #5

Hi magnus,

I am not able too see any logstash config issue.
I will try to give more detail about my ques
my input is
message => field id="0" value="01"/>\n field id="3" value="03"/>\n <field
id="5" value="05"/>\n field id="7" value="07"/>\n <field id="9"
value="09"/>\n /isomsg>\n"

I am applying kv filter here

kv {
source => "message"
trim => """
trimkey => """
field_split => " "
value_split => "="
}

The output I receive is
"id" => [
[0] "0",
[1] "3",
[2] "5",
[3] "7",
[4] "9"
],
"value" => [
[0] "01",
[1] "03",
[2] "05",
[3] "07",
[4] "09"
],

Now I want to write a ruby code where I can extract the id and value wrt to index

The ruby code should be in loop and check , if id[index] == "7"
event['thisvalue'] = value[index]

Means first find the index where id is 7(say indexA is 3) and then extract the value at indexA(which should be 07) and store this value as a field.

For this the code i wrote is

ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == "7"
event['IndexValue'] = i}
"
}
This code should at-least give me index where the value for "id" is 7 which is not happening and I get ruby fail exception as a tag.

"tags" => [
[0] "_rubyexception"
],


(Magnus Bäck) #6

I am not able too see any logstash config issue.

Look harder. There must be something in the log.


(Aditya Srivastava) #7

Which log are you exactly referring to??


(Magnus Bäck) #8

The Logstash log file, typically in /var/log/logstash. If Logstash misbehaves it's the first thing you should check.


(Aditya Srivastava) #9

I have take the logstash zip and unzipped it at a location. I am now running my configuration files from that very location.
bin/logstash -f myfile.conf

So there is no /var/log/logstash which I have. For that matter, I do not see any log file where logstash is dumping any kind of detail.


(Magnus Bäck) #10

If you run Logstash like that it should log to your terminal.

I suspect the configuration you've posted is different from what your running, because I can spot two errors in your Ruby code that would prevent Logstash from even starting.


(Aditya Srivastava) #11

You are correct. It doesnt work for above ruby code.
My logstash config file ran with by commenting few lines.

I took reference from below code written by you for some other post

filter {
ruby {
code => '
ids = event["id"].split(",")
values = event["value"].split(",")
if ids.length == values.length
ids.each_index { |i| event["id#{ids[i]}"] = values[i] }
end
'
}
}

Could you please tell me what is wrong in the ruby code I wrote or help me with ruby code to solve the prob


(Magnus Bäck) #12

There are at least these two problems:

  • You can't use double quotes in the Ruby code since you're using double quotes as the Ruby code delimiter, i.e. replace if times[i] == "7" with if times[i] == '7'.
  • There's an end missing at the end of the whole expression.

(Aditya Srivastava) #13

Thanks for pointing out the mistakes.
ruby {
code => "
times = event['id'].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == '7'
event['IndexValue'] = i}
end
"
}

Now when I run this, I get foll exception

SyntaxError: (ruby filter code):5: syntax error, unexpected tRCURLY
event['IndexValue'] = i}
^
eval at org/jruby/RubyKernel.java:1079
register at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-2.0.3/lib/logstash/filters/ruby.rb:29
each at org/jruby/RubyArray.java:1613
start_filters at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:171
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/pipeline.rb:101
execute at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/agent.rb:165
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:90
call at org/jruby/RubyProc.java:281
run at /data/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.3-java/lib/logstash/runner.rb:95
call at org/jruby/RubyProc.java:281
initialize at /data/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:24

I looked up for the exception, found a solution given by you.

ruby {
code => "
times = event["id"].split(',')
times.each_index { |i| event[i.to_s] = times[i]
if times[i] == '7'
event['IndexValue.to_i'] = i}
end
"
}

when change event['id'] from single quote to double quotes as mentioned above, it gives me foll exception

Error: Expected one of #, {, } at line 25, column 16 (byte 545)

If I comment out the complete ruby code, It works fine. Means again there is some prob with the ruby code.

Plz refer some link as well where I can study more detail on how to use ruby in logstash.


#14

There's a problem here:

times.each_index { |i| event[i.to_s] = times[i]
  if times[i] == '7'
    event['IndexValue'] = i}
  end

Try:

times.each_index { |i| event[i.to_s] = times[i]
  if times[i] == '7'
    event['IndexValue'] = i
  end
}

Or close your brace before (I don't know what you're trying to do)


(Magnus Bäck) #15

Replace

if times[i] == '7'
  event['IndexValue'] = i}
end

with

if times[i] == '7'
  event['IndexValue'] = i
end
}

Plz refer some link as well where I can study more detail on how to use ruby in logstash.

Apart from the ruby filter documentation there isn't much available that applies without reservations to Logstash 2.1.


(Aditya Srivastava) #16

Yes. The solution works and the config file runs properly
But still receiving

"tags" => [
[0] "_rubyexception"

Also this message gets logged on console in red color

Ruby exception occurred: undefined method split' for nil:NilClass {:level=>:error} Ruby exception occurred: undefined methodsplit' for ["0", "3", "5", "7", "9"]:Array {:level=>:error}


(Aditya Srivastava) #17

If possible request you to start fresh as this is getting messy. I will try and frame my query in more clear and structured way now


(Aditya Srivastava) #18

Is this in array format? Can I apply split function on this event.
I feel somehow the ruby exception is because of this array that I am trying to split
Please confirm


(Magnus Bäck) #19

Ruby exception occurred: undefined method split' for nil:NilClass {:level=>:error}

That error indicates that not all your events have an id field. event['id'] evaluates to nil when the field doesn't exist.

Ruby exception occurred: undefined methodsplit' for ["0", "3", "5", "7", "9"]:Array {:level=>:error}

You can split strings but not arrays. Arrays don't need to be split.


(Aditya Srivastava) #20

What should I do then if I want to find the index of "7" in given array??