A massive amount of active generic thread

(liu wei) #1

My nodes recently started to have accumulate amount of active generic threads, growing steadily to 20k+, then the node is crashed.

I tried to look from the hot thread but it's not responding. Any other ways to troubleshoot?

(Mark Walkom) #2

What version are you on?
What other things do you monitor? Anything in logs?

Can you try to check hot threads before things become unresponsive?

(liu wei) #3

Happens both on 1.5 and 1.6. I'm monitoring everything that Marvel sends out.
There's nothing in the log. And yes, I'm planning to dump hot thread info regularly too.

Is there any java tools to analyze this? I'm on windows server 2012

(Mark Walkom) #4

What other things does Marvel show increasing in similar ways?

Can you check _cat/pending_tasks?

(liu wei) #5

that's worrying, i have a long pending task, many of them are
218935 5.1s NORMAL indices_store
What is this?

Some are
220192 2.1s URGENT shard-started

(Mark Walkom) #6

Sounds like your node is overloaded, how much data do you have in it?

(system) #7