I have this pipeline setup for my thesis:
Webserver+rsyslog->Kafka (3 servers: 1xZK, 2xKafka Brokers)->Logstash (2 servers: Both acting as indexers)->ES+Kibana (both on the one server). Everything running on AWS. Currently handling 6500 messages / sec.
From what I read and from my experiments the suggested bottleneck is with ES. I was reading a tutorial the other day that stated to set up 3 master nodes, 2 client nodes, 2 data nodes etc but I could not get it working. I don't need any fallback or scaling. my sole focus is on just increasing throughput.
What is the simplest cluster I can make that will increase the messages / second that ES can handle?
And could someone recommend a tutorial that helps walk me through the process?
Thanks in advance.