Multiple pipelines running on same machine

[adityaPsl (Aditya)]

Hello Team,

I am facing issues with logstash pipelines filter section i am using multiple filters like grok, elasticsearch translate, ruby etc in combination, The issue is the data of some other record processed at the around the same time overwrites the field values and sometimes adds exception in tags but when the records processed individually filter section works fine with no exceptions and data overwritten.
does multiple pipelines use the same JVM, are ruby variables shared between two events.
are there any tools recommended to debug the issue.

ELK Version : 7.4.2

Thank you,
Aditya

[Badger]

@adityaPsl

does multiple pipelines use the same JVM, are ruby variables shared between two events

Yes, pipelines in the same logstash instance run in the same JVM. Where variables in ruby filters are shared depends on the scope with which you declare them. Global variables ($name) are shared globally. You should never need that. Class variables (@@name) are shared across all ruby filters. It is not every year I need a class variable and I write a lot more ruby filters that most folks. Instance variables (@name) are shared by different events within a single ruby filter. Sometimes I use instance variables but most of the time I just use regular variables (name) that just exist for the duration of the execution of the code block for an event.

[adityaPsl (Aditya)]

Thank you Badger,

in my pipeline i have used simple names for example

ruby  {

         code => "
                    #field2 value i am getting from grok filter
                    field = event.get('field2');
                    event.set('custom_name',filed[0..10])
          "

}

but with multiple pipelines i can see field2 value is different but the value getting assigned to field 'custom_name' is different its getting overwritten with some other records value.

Not sure why, any thoughts on this please.

Thank you,
Aditya

[Badger]

@adityaPsl

any thoughts on this please.

I cannot think of anything that could cause that.

[adityaPsl (Aditya)]

Thank you Badger.

any tools by which I can debug or get to the root cause.

Thank you,
Aditya

[adityaPsl (Aditya)]

Hello Badger,

will it make any difference if the variable name starts with capital letter
ruby {

     code => "
                #field2 value i am getting from grok filter
                Field = event.get('field2');
                event.set('custom_name',Field[0..10])
      "

}

Thank you,
Aditya

[Badger]

Well, if it starts with a capital letter it is a constant, not a variable, but since you are not trying to modify it that should not be a problem.

[adityaPsl (Aditya)]

Right, and even if it is a constant it should get updated with each event and not like a shared constant between events.
Is my understanding correct?

Thank you,
Aditya

[Badger]

@adityaPsl

Is my understanding correct?

I believe so.

[adityaPsl (Aditya)]

Thank you Badger,
Sorry to ask again.
How can I go to the root cause and fix, any tools or debugging options.
Thank you,
Aditya

[adityaPsl (Aditya)]

Hello Team,

I tried to do the simple test

Crawl the data from csv and copy one field into another by using ruby variable .but looks like data is getting overwritten from somewhere.
I followed the below steps

1. created 2 pipeline with same filter and input section only chnaged the index in output section

Here is the code pipeline1

 input {

 file {
    path => "<file_path`Preformatted text`>/*.csv"
    start_position => "beginning"
  }

}

filter {

	csv {
		  separator => ","
		  columns => ["id,first_name,last_name,email,gender,ip_address"]
	}
	
	
	
	
	ruby {
	
		code => "
		
			TestVar = event.get('column6');
			event.set('ipcpy',TestVar)
		
	
		"
	
	}
	
	ruby {
	
		code => "
		
			if event.get('ipcpy') == event.get('column6')
				event.set('cpystat','1')
			else
			event.set('cpystat','0')
			end
		
	
		"
	
	}

}


output {


	elasticsearch {
     hosts => "http://localhost:9200"
     index => "testdata1"
  }

}

Pipeline2

input {

 file {
    path => "<FilePath>/*.csv"
    start_position => "beginning"
  }

}

filter {

	csv {
		  separator => ","
		  columns => ["id,first_name,last_name,email,gender,ip_address"]
	}
	
	
	
	
	ruby {
	
		code => "
		
			TestVar = event.get('column6');
			event.set('ipcpy',TestVar)
		
	
		"
	
	}
	
	ruby {
	
		code => "
		
			if event.get('ipcpy') == event.get('column6')
				event.set('cpystat','1')
			else
			event.set('cpystat','0')
			end
		
	
		"
	
	}

}


output {


	elasticsearch {
     hosts => "http://localhost:9200"
     index => "testdata"
  }

}

I ran both the pipelines at the same time from different command prompts.
and checked the variable cpystat , from code cpystat is set to '1' if copied field and origional field is same else it is set to '0' and surprisingly i found that around 170 records did not copy the field correctly.

could you please help me understand what is the issue here.

Thank you for your help and support,

Thank you,
Aditya

[adityaPsl (Aditya)]

Looks like the issue here is the variable case, if i change variable from TestVar to testVar then its not causing any issues, even with multiple pipelines.

still i would like to understand what is the issue here with capital case variable name in logstash ruby code.

Thank you,
Aditya

[Badger]

The pipeline configurations you show supply column names for six columns, so if you were running them then event.get('column6') would return nil. So it appears you are not doing what you say you are doing.

[adityaPsl (Aditya)]

Ummm...i also tried that, but for some reason, it is considering that column list as one field (may be some syntax issue) anyways , that is okay for now.
and i am getting data in event.get('column6')

Please see the screen shot attached in earlier post , you can see column6 has the value(ip address) and ipcpy field is copy of that field

here is the one more screenshot for reference.

Thank you,
Aditya

[Badger]

My mistake. I misread what you posted. As you say, you have the syntax wrong

columns => ["id,first_name,last_name,email,gender,ip_address"]

should be

columns => ["id", "first_name", "last_name", "email", "gender", "ip_address"]

[adityaPsl (Aditya)]

Thank you Badger,
Yeah, my focus was on copying the field with ruby variable and multiple pipelines, so ignored that. Sorry.
but still struggling to understand what was wrong with captial letter. can you suggest any documentation please.

Thank you,
Aditya

[Badger]

In Ruby, a "variable" that starts with an uppercase letter is a constant, not a variable. That said, ruby contstants are mutable, so I would not expect it to actually matter.