Multiple pipelines running on same machine

Hello Team,

I am facing issues with logstash pipelines filter section i am using multiple filters like grok, elasticsearch translate, ruby etc in combination, The issue is the data of some other record processed at the around the same time overwrites the field values and sometimes adds exception in tags but when the records processed individually filter section works fine with no exceptions and data overwritten.
does multiple pipelines use the same JVM, are ruby variables shared between two events.
are there any tools recommended to debug the issue.

ELK Version : 7.4.2

Thank you,
Aditya

Yes, pipelines in the same logstash instance run in the same JVM. Where variables in ruby filters are shared depends on the scope with which you declare them. Global variables ($name) are shared globally. You should never need that. Class variables (@@name) are shared across all ruby filters. It is not every year I need a class variable and I write a lot more ruby filters that most folks. Instance variables (@name) are shared by different events within a single ruby filter. Sometimes I use instance variables but most of the time I just use regular variables (name) that just exist for the duration of the execution of the code block for an event.

1 Like

Thank you Badger,

in my pipeline i have used simple names for example

ruby  {

         code => "
                    #field2 value i am getting from grok filter
                    field = event.get('field2');
                    event.set('custom_name',filed[0..10])
          "

}

but with multiple pipelines i can see field2 value is different but the value getting assigned to field 'custom_name' is different its getting overwritten with some other records value.

Not sure why, any thoughts on this please.

Thank you,
Aditya

I cannot think of anything that could cause that.

Thank you Badger.

any tools by which I can debug or get to the root cause.

Thank you,
Aditya

Hello Badger,

will it make any difference if the variable name starts with capital letter
ruby {

     code => "
                #field2 value i am getting from grok filter
                Field = event.get('field2');
                event.set('custom_name',Field[0..10])
      "

}

Thank you,
Aditya

Well, if it starts with a capital letter it is a constant, not a variable, but since you are not trying to modify it that should not be a problem.

Right, and even if it is a constant it should get updated with each event and not like a shared constant between events.
Is my understanding correct?

Thank you,
Aditya

I believe so.

Thank you Badger,
Sorry to ask again.
How can I go to the root cause and fix, any tools or debugging options.
Thank you,
Aditya