I can barely get above 6,000 EPS/s with the filebeat and googlecloud module. The best I did was 6,487 EPS/s. Server CPU's utilization was about 25%
I started with the blog post: https://www.elastic.co/blog/how-to-tune-elastic-beats-performance-a-practical-example-with-batch-size-worker-count-and-more. Next, I set up the following configs (after testing many different ones first):
#set queue.mem.events to 2 * workers * batch size
#set queue.mem.flush.min_events to batch size
#this is optional, I tested it with 0, 5, and finally with 9
I then looked at the googlecloud module (pubsub connection), and I added the following configs:
For the googlecloud module, I also tested different configs (large numbers or even -1), some of that was described in here https://github.com/googleapis/google-cloud-go/wiki/Fine-Tuning-PubSub-Receive-Performance#subscriptionreceivesettingsmaxoutstandingmessages).
And, I can't get anything above 6,487 EPS/s, any thoughts?
Is there a way to measure the filebeat performance after deploying each configuration change other than looking into the Kibana metric dashboard?