I have found some posts on this before. But, have not got a definitive answer with respect to Beats 6.x as spool_size has been removed.
The following tests have been performed on
Elasticsearch 6.2.4 with 16GB Heap
Filebeat config with File Output gives 80,000 events/s
Filebeat config with elasticsearch output gives 14,000 events/s
The elasticsearch indexing rate is 13,800 events/s and this seems to the bottleneck.
What i dont understand is Elasticsearch CPU Utilization is 10% and JVM Heap Used is 6GB/16GB. Then why is the indexing rate still so low? What other factors should we consider to stress the elasticsearch system?
Any suggestions on improving this performance would be highly appreciated.