Thankyou @spinscale.
First, you may want to have dedicated master nodes (own processes/instances), so that nodes doing the indexing work do not need to deal with master node tasks and vice versa.
Yes, all three are running on independent hosts, with its own process/instance.
Also do not use dedicated master nodes to send indexing data two, always send to the master nodes (note: clients can handle this automatically if sniffing is enabled).
Sorry I did not understand this (pardon me am a newbie to elastic). Does this mean that I should setup logstash to point to the master node only, and then master node will take care of the rest?
You can run logstash and elasticsearch on the same nodes, but this also implies they will potentially steal each other resources, meaning that performance issues will be hard to debug.
Thanks for the suggestion, totally makes sense.
The elasticsearch rollover API is used to create a new index, but it does not delete old ones. You should use something like curator1 to do those cleanup tasks.
How does kibana adapt to the changing indices. Say I use rollover API and curator to clean up old one, and all my dashboard/visualization are linked to indexA, is there a way to automatically link kibana dashboards to the rolled-over indexA_1 ?