I have been testing Elasticsearch & Kibana for some time. I need a cluster that can process 40k events per second and search inside the last week of data under one minute of search time. My cluster consists of 9 data nodes, 3 master nodes and one client node. I installed the X-Pack today so I can use the profiler and monitoring software.
The cluster is meant to store and analyze Netflow and IPFIX data. I currently ingest around 5000 events per second, which should be really easy. However, searching this data is terribly slow. I was hoping that the monitoring could help me point out what is going wrong. Something inmediately raised a red flag for me.
I don't have 209 GB of available memory for no reason! Why does it not use more memory? I have tried all kinds of basic tweaks and optimizations that are available on the Internet for Elasticsearch clusters. However, I am probably missing something. Logstash also was way too slow, where others could get better performance.
Any ideas on why my Elasticsearch cluster uses barely 10% of my total available memory?