How to write while Loops inside plugins or filters?

Elastic Stack Logstash
1

Hi,

Is there any way to write loops inside filters or outputs plugins?

For a certain message

{"time"=100,"name"="test"}

I want to implement something like below:

c=0,
while c<=10
time=time+c
c=c+1

if c=10
reset to c=0

For every message the time needs to get updated until the count reaches 10 and the count needs to reset to 0.
where can i write this logic ? Filters ? outputs ?
I need every message i send to the input filter to get this udpdated before it transforms as output.

2

You could implement that in a ruby filter or an aggregate filter.

3

I am new to ruby and I did write code to do it.

ruby {
    code => "event.set('timestamp', event.get('timestamp_object').to_i * 1000);
    event.set('time',event.get('timestamp_object').to_i*100000000);
    count=0
    while count<=10 do
    puts count
    time=(time+count)
    if count==10
    count=0
    puts 'resetting count to 0'
    end
    end"
  }

I did see the below error though

[2019-06-05T16:58:38,521][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `+' for nil:NilClass
[2019-06-05T16:58:38,523][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `+' for nil:NilClass

and this as the message:

{
          "tags" => [
        [0] "_grokparsefailure",
        [1] "_rubyexception"
    ],
    "@timestamp" => 2019-06-05T20:58:38.226Z,
       "message" => "/-/depd-1.1.2.tgz|HTTP/1.1|200|9026",
          "host" => "SI-M-CG5",
      "@version" => "1",
          "time" => 0,
          "path" => "/testing/test-arti.log",
     "timestamp" => 0
}

Do you know what might be causing this or if my code is faulty ?

4

You never initialized the time variable, so it is nil, and you cannot add to nil.

5

I was hoping this would be sufficient to intialize it

6

No, that adds a field to the event, it does not create a ruby variable. You could add

time = event.get('time')

after that statement.

7

Thanks ! that did fix the error but i think my code is written wrong for the requirements. It's actually going to an infinite loop when i want to execute and update only once for every execution and i am getting a grok parse failure

code => "event.set('timestamp', event.get('timestamp_object').to_i * 1000);
    event.set('time',event.get('timestamp_object').to_i*100000000);
    count=0
    if count<=10
    count=count+1
    puts count
    time=event.get('time')
    puts 'time'
    time=(time+count)
    puts time
    end
    if count==10
    count=0
    puts 'resetting count to 0'
    end"

and this is the ouptut

1
time
1
{
"message" => "-npm-npm/gzip-size/-/gzip-size-3.0.0.tgz|HTTP/1.1|200|2015",
"tags" => [
[0] "_grokparsefailure"
],
"timestamp" => 0,
"@version" => "1",
"host" => "SI-M-C5",
"@timestamp" => 2019-06-05T22:44:00.693Z,
"time" => 0,
"path" => "/testing/test-arti.log"
}

8

I do not really understand the requirements of your use case but this might demonstrate why this is really not a good fit for logstash.

This code takes the time field from an event and adds a number from 0 to 10 to it.

input { generator { count => 100 lines => [ '{ "time": 100}' ] } }
filter {
    json { source => "message" }
    ruby {
        init => '
            @c = 0
        '
        code => '
            event.set("timex", @c + event.get("time"))
            @c += 1
            if @c >= 10
                @c = 0
            end
        '
    }
}

output { stdout { codec => plain { format => "%{timex}
" } } }

If you run that with "-w 3 -b 3" then it results in a stream of numbers that starts with

100
101
102
106
107
108
109
100
101
102
103
104
105
106
107
103
104

The reason I set "-b 3" is so that I do not have to post a thousand lines of data to demonstrate the issue of different threads interacting. If you set "-w 1" then the result does cycle between 100 and 109, but the solution does not scale.

9

This might not be the best approach and i am trying to avoid an issues with InfluxDB mentioned here. The only work arounds are to use uuid's as another field or to modify values for fields so that the timestamp values are not similar for at least 5-10 sec.
I thought running through a loop an adding it to timestamps is the best way.

Do you think if writing a shell script to update timestamp values for every record would help ?

10

Is it possible to update the events before they reach filters ? someway after reading the messages through input plugins and before filters ?

11

I tried something similar to this but I get "Ruby exception occurred: undefined local variable or method `count' for #LogStash::Filters::Ruby:0x138f09da" error.

code:

ruby {
    init => "count = 0"
    code => "event.set('timestamp', event.get('timestamp_object').to_i * 1000);
    event.set('time',event.get('timestamp_object').to_i*100000000);
        if count<=10
        count=count+1
        puts count
        event.set('time',event.get('time')+count)
        puts time
        end
        if count==10
        count=0
        puts 'resetting count to 0'
        end"
  }
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.