Hi,
I have created a elasticsearch cluster with 4 data nodes (each 32 GB ram, 8 core CPU, 500 GB hard-disk) , 3 master nodes (each 32 GB ram, 8 core CPU, 50 GB hard-disk) and 2 load balancer (each 32 GB ram, 8 core CPU, 50 GB hard disk ).
There would be on an average 500 GB to 1 TB events per day from about 5000 devices (which would increase to 25000 devices within 6 month). I have created 3 indexer each for linux, windows and network devices (which is created on per month bases).
What would be the number of shards in this environment? What should be the configuration of elasticsearch to avoid any bottleneck?
Currently, I am facing a lag (3-4 hours which is increasing when any query in applied at kibana) in receiving the logs at Kibana. I have specified number of shards as 5 with no replicas.
Moreover, when I see through the API , /_cat/nodes?v , the ram_percent of the data nodes is above 80 percent with 0.05 load.