Ruby script with dynamic variables in logstash

Elastic Stack Logstash
1

Is it possible to have a dynamic variables in ruby that changes based on the input values I am getting from the inputfile?

For example I have 5 input values which the attribute linked, if the value of link is 'connected' then I will need to increase the variable otherwise nothing. And for each value I will need to assign in a new attribute the current value of the variable.

if the input is:

  1. username = test1, linked = disable, ...
  2. username = test2, linked = connected, ...
  3. username = test3, linked = connected, ...
  4. username = test4, linked = disable, ...
  5. username = test5, linked = connected, ...

the output should be:

  1. username = test1, linked = disable, status_id = 0 ...
  2. username = test2, linked = connected, status_id = 1, ...
  3. username = test3, linked = connected, status_id = 2, ...
  4. username = test4, linked = disable, status_id = 2, ...
  5. username = test5, linked = connected, status_id = 3, ...
2

Can you explain the logic behind this output? Is there any typo on it?

It seems that you want to increment the count for each event, for example, the first event with disabel would get the the status_id of 0, shouldn't the next event get the status_id of 1? On the output you shared you jump to 2, also, the first event of connected starts at 1, not 0.

So, assuming that this was a typoe, you may be able to do that with the following filters:

filter {
    if [status] == "disable" {
        ruby {
            init => '@@count_disabled = 0'
            code => '
                event.set("status_id", @@count_disabled)
                @@count_disabled = @@count_disabled + 1
            '
        }
    }
    if [status] == "connected" {
        ruby {
            init => '@@count_connected = 0'
            code => '
                event.set("status_id", @@count_connected)
                @@count_connected = @@count_connected + 1
            '
        }
    }
}

Considering that you have the linked value on a field named status, this will increment the count of the status_id for each event based on the value of this field.

But this has some catchs, one you need to run it with just one worker, which means that logstash will use just one cpu core for this pipeline, if you restart or stop logstash, you will lose the count of the status_id.

1 Like
3

Thank you for the answer.

The idea here is to only increase the counter when the link is connected otherwise it should stay the same.
This is why for test1 it is disable so it is 0,
for test2 it is connected so it is 1
for test3 it is connected so it is 2
for test4 it is disable so it is 2
and finally for test5 it is connected so it is 3.

In the example above the @@count_connected is stored as external value?

How can I set logstash with one worker?

And in case of stopping logstash before it is finish you mean?
Because if I will restart it will start the counting again correct?

4

Then you would need only the conditional checking if the value is connected.

It is stored in Logstash memory.

You can set pipeline.workers: 1 in logstash.yml or pipelines.yml, or if you are running using the command line you need to use -w 1.

If the logstash process stops for any reason, it will start counting from 0 again.

In Logstash every event is independent from each other and while sometimes you can correlate them during the ingestion, it may be not the best tool for this because of its limitation.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.