Logstash is slow for mulitiple pipelines

Hi There,
I am using two pipelines to process my data coming from same server but different set of files. let's say A and B. A has a very high tps and it goes over 1000EPS(events per sec) B has 40EPS max. They push data to two separate indexes at elastisearch.
I see a big delay on pipeline B when events are visible on kibana dashboard via elasticsearch. I don't see a delay on pipeline A though!
As the pipeline A data isn't that critical, I tried to slow down it's processing by
changing pipeline A settings ,

increase the pipeline.batch.delay
pipeline.workers to 1
reducing pipeline.batch.size

while pipeline B settings were left at default values.

The result was pipeline A EPS came down to 50 and pipeline B also went below 1.

I didn't see memory or cpu going above 50%.

can someone help me how to change the pririty of the pipelines when they handle data?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.