What does Warm-node do after Curator running?

[af615dbd55cac2cacf32 (Jason J Youm)]

Hello,

I posted a similar post yesterday but nobody replied yet, so I am posting this topic as a simple question.

Possible to see what each nodes in cluster are doing

Hello I was wondering if there are any ways to see what each cluster node is exactly doing. My Elasticsearch cluster was set as below and situation is here. We installed 9nodes(2master:1active, 1client, 3hot and 3warm nodes). Current TPS is about 20k and we are adding 3k TPS everyday. Hot nodes store only 1day data(daily created indices) and Curator is runs everyday to move indices to warm nodes. The problem i need some advises here is about the warm nodes. CPU usage on warm nodes are gettin…

• What does Warm-node do after Curator running? Are they performing RE-Indexing?

• Warm-nodes keep holding CPU usage after old indices came via Curator? CPU usage is getting higher based on the indices size increase from Hot-nodes, and never goes down.

Please advise!

[Christian_Dahlqvist]

What is the heap usage on the warm nodes like? Do you see any evidence of long or frequent GC in the Elasticsearch logs that could explain CPU usage? Are you querying the data on the warm nodes?

[af615dbd55cac2cacf32 (Jason J Youm)]

Thank you for your replies.

I am not sure if i am able to answer for your question correctly but willing to try.

Heap usage was about 100% when i checked with _cat query.

Can you please advise how to check GC status in the log? Fyi, my log level is set as default so i assume current level is INFO. Is it enough to see the GC status in the log?

Sorry the datacenter where i worked is really strick and cannot access from outside. I can provide more detail info next monday.

[Christian_Dahlqvist]

If it was close to 100%, it sounds like you are suffering from heap pressure. There are a few things you should look at:

• Have you optimised your mappings in order to minimise the amount of field data being used? Default dynamic mapping maps string fields as both text and keyword. If you are only going to run aggregations against them and not free-text search, you can reduce heap usage significantly by mapping them as keyword only.
• Having lots of small indices and shards can also be inefficient and drive up heap usage. have a look at this blog post for guidance on shards and sharding practices.

If you have already applied these recommendations, you may simply be at the point where you need to scale up or out.

[af615dbd55cac2cacf32 (Jason J Youm)]

Yes we already applied both recommendations. I will take a look again if there are more thing to do for #2. Also we are considering scale out next month.

Thank you a lot!!