I was wondering if there are any ways to see what each cluster node is exactly doing.
My Elasticsearch cluster was set as below and situation is here.
We installed 9nodes(2master:1active, 1client, 3hot and 3warm nodes). Current TPS is about 20k and we are adding 3k TPS everyday. Hot nodes store only 1day data(daily created indices) and Curator is runs everyday to move indices to warm nodes.
The problem i need some advises here is about the warm nodes. CPU usage on warm nodes are getting higher everyday; while Curator is running, the warm nodes getting higher CPU usage then never go down. I have no idea what warm nodes are doing after Curator task is done.
Is there any commands or ways to see what the warm nodes are doing after Curator run? CPU usage is getting higher and never going down, but this higher CPU usage i am observing only after Curator is running.
Fyi, we need to run curator everyday due to disk space issue on hot nodes.