I ran "top -b -n 1 -c -H" every 10Sec and this is the first result after it got stuck (and i saw it even before in previous cases that elasticsearch is the one that consuming cpu with htop):
20240116-112256
top - 11:22:57 up 369 days, 20:52, 3 users, load average: 36.34, 31.48, 30.92
Threads: 6661 total, 15 running, 6646 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.6 us, 1.6 sy, 0.0 ni, 76.6 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 79100940+total, 10132932 free, 32252688+used, 45834963+buff/cache
KiB Swap: 0 total, 0 free, 0 used. 46524675+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
129219 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:20.12 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129407 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.42 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129408 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.76 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129398 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.08 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129399 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.44 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129400 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129401 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.84 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129402 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.45 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129403 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.20 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129404 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.36 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129405 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129406 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128633 elastic+ 20 0 1379.8g 45.5g 9.2g S 60.7 6.0 141:33.08 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
166722 elastic+ 20 0 1999.5g 47.4g 8.7g R 57.1 6.3 2:15.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167736 root 20 0 168936 9052 1616 R 50.0 0.0 0:00.26 top -b -n 1 -c -H
167608 elastic+ 20 0 1819.5g 41.6g 5.8g S 46.4 5.5 0:16.58 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129030 elastic+ 20 0 1379.8g 45.5g 9.2g R 35.7 6.0 21:56.54 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
138637 elastic+ 20 0 1999.5g 47.4g 8.7g S 21.4 6.3 897:46.60 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141011 elastic+ 20 0 1999.5g 47.4g 8.7g S 21.4 6.3 51:43.26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141004 elastic+ 20 0 1999.5g 47.4g 8.7g S 17.9 6.3 55:37.32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141912 elastic+ 20 0 1999.5g 47.4g 8.7g S 14.3 6.3 282:18.38 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143046 elastic+ 20 0 1999.5g 47.4g 8.7g S 14.3 6.3 280:08.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147525 elastic+ 20 0 1700.6g 43.2g 5.1g S 10.7 5.7 141:24.04 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142866 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 284:58.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142921 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 191:43.88 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143223 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 281:28.91 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143233 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 281:18.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129745 elastic+ 20 0 1819.5g 41.6g 5.8g S 7.1 5.5 8:00.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130669 elastic+ 20 0 1819.5g 41.6g 5.8g S 7.1 5.5 33:54.96 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
137756 elastic+ 20 0 1700.6g 43.2g 5.1g S 7.1 5.7 590:22.48 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142949 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 281:56.28 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143247 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 281:11.15 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143287 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 283:22.36 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143283 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 0:09.26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
166654 elastic+ 20 0 1379.8g 45.5g 9.2g S 3.6 6.0 0:32.56 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130628 elastic+ 20 0 1819.5g 41.6g 5.8g S 3.6 5.5 33:56.09 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147527 elastic+ 20 0 1700.6g 43.2g 5.1g S 3.6 5.7 138:26.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141009 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 41:24.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141080 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 48:11.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142977 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 281:06.01 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147037 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 189:04.43 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
92453 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 2:18.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
122450 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 1:00.61 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131861 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.72 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131870 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.68 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131872 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.98 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131873 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:20.21 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131875 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:19.21 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131881 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:19.33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
133167 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:15.76 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
144168 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:08.32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
144584 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:09.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147680 elastic+ 20 0 2042.5g 44.8g 6.9g S 3.6 5.9 178:21.45 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
In addition, I tried to put the jstack logs that I sent here and I took the first log after it got stuck (the log from the followers check), and put it in a website that analyze it, and got this:
And also this:
Do you think that it might be right or it just follows simple thumb rules to analyze it?