Hi There,
I am using two pipelines to process my data coming from same server but different set of files. let's say A and B. A has a very high tps and it goes over 1000EPS(events per sec) B has 40EPS max. They push data to two separate indexes at elastisearch.
I see a big delay on pipeline B when events are visible on kibana dashboard via elasticsearch. I don't see a delay on pipeline A though!
As the pipeline A data isn't that critical, I tried to slow down it's processing by
changing pipeline A settings ,
increase the
pipeline.batch.delay
pipeline.workersto 1
reducingpipeline.batch.size
while pipeline B settings were left at default values.
The result was pipeline A EPS came down to 50 and pipeline B also went below 1.
I didn't see memory or cpu going above 50%.
can someone help me how to change the pririty of the pipelines when they handle data?