# Multiple pipelines running on same machine

**URL:** <https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217>\
**Category:** Logstash\
**Created:** [March 26, 2020, 2:40pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217 "2020-03-26T14:40:56Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [March 26, 2020, 2:40pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/1 "2020-03-26T14:40:56Z")

</div>

Hello Team,

I am facing issues with logstash pipelines filter section i am using multiple filters like grok, elasticsearch translate, ruby etc in combination, The issue is the data of some other record processed at the around the same time overwrites the field values and sometimes adds exception in tags but when the records processed individually filter section works fine with no exceptions and data overwritten.  
does multiple pipelines use the same JVM, are ruby variables shared between two events.  
are there any tools recommended to debug the issue.

ELK Version : 7.4.2

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 26, 2020, 5:39pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/2 "2020-03-26T17:39:00Z")

</div>

> [@adityaPsl](#):
>
> does multiple pipelines use the same JVM, are ruby variables shared between two events

Yes, pipelines in the same logstash instance run in the same JVM. Where variables in ruby filters are shared depends on the scope with which you declare them. Global variables ($name) are shared globally. You should never need that. Class variables (@@name) are shared across all ruby filters. It is not every year I need a class variable and I write a lot more ruby filters that most folks. Instance variables (@name) are shared by different events within a single ruby filter. Sometimes I use instance variables but most of the time I just use regular variables (name) that just exist for the duration of the execution of the code block for an event.

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [March 26, 2020, 6:09pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/3 "2020-03-26T18:09:54Z")

</div>

Thank you Badger,

in my pipeline i have used simple names for example

```
ruby {

         code => "
                    #field2 value i am getting from grok filter
                    field = event.get('field2');
                    event.set('custom_name',filed[0..10])
          "

}

```

but with multiple pipelines i can see field2 value is different but the value getting assigned to field 'custom\_name' is different its getting overwritten with some other records value.

Not sure why, any thoughts on this please.

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 26, 2020, 6:39pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/4 "2020-03-26T18:39:31Z")

</div>

> [@adityaPsl](#):
>
> any thoughts on this please.

I cannot think of anything that could cause that.

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [March 27, 2020, 3:24am UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/5 "2020-03-27T03:24:47Z")

</div>

Thank you Badger.

any tools by which I can debug or get to the root cause.

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 6, 2020, 11:57am UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/6 "2020-04-06T11:57:16Z")

</div>

Hello Badger,

will it make any difference if the variable name starts with capital letter  
ruby {

```
     code => "
                #field2 value i am getting from grok filter
                Field = event.get('field2');
                event.set('custom_name',Field[0..10])
      "

```

}

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 6, 2020, 2:10pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/7 "2020-04-06T14:10:32Z")

</div>

Well, if it starts with a capital letter it is a constant, not a variable, but since you are not trying to modify it that should not be a problem.

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 6, 2020, 2:32pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/8 "2020-04-06T14:32:56Z")

</div>

Right, and even if it is a constant it should get updated with each event and not like a shared constant between events.  
Is my understanding correct?

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 6, 2020, 4:11pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/9 "2020-04-06T16:11:58Z")

</div>

> [@adityaPsl](#):
>
> Is my understanding correct?

I believe so.

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 6, 2020, 6:09pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/10 "2020-04-06T18:09:09Z")

</div>

Thank you Badger,  
Sorry to ask again.  
How can I go to the root cause and fix, any tools or debugging options.  
Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 24, 2020, 8:30am UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/11 "2020-04-24T08:30:55Z")

</div>

Hello Team,

I tried to do the simple test

Crawl the data from csv and copy one field into another by using ruby variable .but looks like data is getting overwritten from somewhere.  
I followed the below steps

1. created 2 pipeline with same filter and input section only chnaged the index in output section

Here is the code pipeline1

```
 input {

 file {
    path => "<file_path`Preformatted text`>/*.csv"
    start_position => "beginning"
  }

}

filter {

	csv {
		  separator => ","
		  columns => ["id,first_name,last_name,email,gender,ip_address"]
	}
	
	
	
	
	ruby {
	
		code => "
		
			TestVar = event.get('column6');
			event.set('ipcpy',TestVar)
		
	
		"
	
	}
	
	ruby {
	
		code => "
		
			if event.get('ipcpy') == event.get('column6')
				event.set('cpystat','1')
			else
			event.set('cpystat','0')
			end
		
	
		"
	
	}

}

output {

	elasticsearch {
     hosts => "http://localhost:9200"
     index => "testdata1"
  }

}

```

Pipeline2

```
input {

 file {
    path => "<FilePath>/*.csv"
    start_position => "beginning"
  }

}

filter {

	csv {
		  separator => ","
		  columns => ["id,first_name,last_name,email,gender,ip_address"]
	}
	
	
	
	
	ruby {
	
		code => "
		
			TestVar = event.get('column6');
			event.set('ipcpy',TestVar)
		
	
		"
	
	}
	
	ruby {
	
		code => "
		
			if event.get('ipcpy') == event.get('column6')
				event.set('cpystat','1')
			else
			event.set('cpystat','0')
			end
		
	
		"
	
	}

}

output {

	elasticsearch {
     hosts => "http://localhost:9200"
     index => "testdata"
  }

}

```

I ran both the pipelines at the same time from different command prompts.  
and checked the variable cpystat , from code cpystat is set to '1' if copied field and origional field is same else it is set to '0' and surprisingly i found that around 170 records did not copy the field correctly.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/f/3f88507c4bb2ef203bd029c259b1aab12f33afa0.png)

could you please help me understand what is the issue here.

Thank you for your help and support,

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 24, 2020, 9:11am UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/12 "2020-04-24T09:11:48Z")

</div>

Looks like the issue here is the variable case, if i change variable from TestVar to testVar then its not causing any issues, even with multiple pipelines.

still i would like to understand what is the issue here with capital case variable name in logstash ruby code.

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 24, 2020, 4:49pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/13 "2020-04-24T16:49:55Z")

</div>

The pipeline configurations you show supply column names for six columns, so if you were running them then event.get('column6') would return nil. So it appears you are not doing what you say you are doing.

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 24, 2020, 5:28pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/14 "2020-04-24T17:28:03Z")

</div>

Ummm...i also tried that, but for some reason, it is considering that column list as one field (may be some syntax issue) anyways , that is okay for now.  
and i am getting data in event.get('column6')

Please see the screen shot attached in earlier post , you can see column6 has the value(ip address) and ipcpy field is copy of that field

here is the one more screenshot for reference.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/0/d0327045cec4fa874506dbde3a84f747a875abd1.png)

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 24, 2020, 5:33pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/15 "2020-04-24T17:33:20Z")

</div>

My mistake. I misread what you posted. As you say, you have the syntax wrong

```
columns => ["id,first_name,last_name,email,gender,ip_address"]

```

should be

```
columns => ["id", "first_name", "last_name", "email", "gender", "ip_address"]
```

---

<div class="post-metadata">

**Author:** ![adityaPsl](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@adityaPsl](https://discuss.elastic.co/u/adityaPsl)\
**Post date:** [April 24, 2020, 5:37pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/16 "2020-04-24T17:37:14Z")

</div>

Thank you Badger,  
Yeah, my focus was on copying the field with ruby variable and multiple pipelines, so ignored that. Sorry.  
but still struggling to understand what was wrong with captial letter. can you suggest any documentation please.

Thank you,  
Aditya

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 24, 2020, 5:41pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/17 "2020-04-24T17:41:26Z")

</div>

In Ruby, a "variable" that starts with an uppercase letter is a constant, not a variable. That said, ruby contstants are mutable, so I would not expect it to actually matter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2020, 5:41pm UTC](https://discuss.elastic.co/t/multiple-pipelines-running-on-same-machine/225217/18 "2020-05-22T17:41:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.