My nodes recently started to have accumulate amount of active generic threads, growing steadily to 20k+, then the node is crashed.
I tried to look from the hot thread but it's not responding. Any other ways to troubleshoot?
What version are you on?
What other things do you monitor? Anything in logs?
Can you try to check hot threads before things become unresponsive?
Happens both on 1.5 and 1.6. I'm monitoring everything that Marvel sends out.
There's nothing in the log. And yes, I'm planning to dump hot thread info regularly too.
Is there any java tools to analyze this? I'm on windows server 2012
What other things does Marvel show increasing in similar ways?
Can you check _cat/pending_tasks?
that's worrying, i have a long pending task, many of them are
218935 5.1s NORMAL indices_store
What is this?
Some are
220192 2.1s URGENT shard-started
Sounds like your node is overloaded, how much data do you have in it?