Speed up the pipeline

Elastic Stack Elasticsearch
1

I have set up of 3 machines:

  1. Single ES process setup with 20gb heap size and 16 processors
  2. 2 logstash processes each with 20gb heap size connecting to the same ES on machine 1
  3. filebeat pumping data to the 2 logstash processes on machine 2

Increasing the heap size on ES fron 2gb default to 20gb did not speed up the pipeline. Increasing the number of logstash from 1 to 2 did not either. Splitting the log file into multiple smaller parts to increase filebeat parallelism didn't help. Neither did increase the bulk_max_size to 8192. Any suggestions on how I can speed up the pipeline.

The interesting part is the CPU usage is going upto 6% at max. RAM usage is not too high either. The available network bandwidth is not being used fully either. So, all available resources are under utilized, yet the pipeline is getting any faster. What am I missing?

2

What about disk I/O and iowait?

3

I/O wait is 0.0 over the period of transfer on all three machines.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.