Hey! I have an elastic cluster (version 8.11.1, upgraded from 8.5.3 but the problem is before the upgrade) with 10 datanode physical servers that is unstable (node are disconnecting and connecting automaticly) with two reasons:
"reason: followers check retry count exceeded"
"reason: disconnected"
After reading this , I tried changing the logging (by whats written in the disconnected chapter) and didnt see something interesting, and then tried to read whats in the "follower check retry count exceeded", changed net.ipv4.tcp_retries2 to recommended value, gc looks fine, didnt really see evidence for gc pauses, didnt really succeeded to see something in Packet captures (I would like for help if it seem like the right thing to do), and right now i cant produce jstack logs because the cluster is very unstable and every time the before the problem the node has high cpu (although it started even when the cpu was fine).
For a start you'll need to share all the logs mentioned in the troubleshooting docs.
This doesn't make sense, jstack will work no matter whether the cluster is stable or not.
Hey!
(by the way, I have 10 servers, and on each server I have 8 node as the number of disks on the server)
The logs are in: https://github.com/elasticexpert/elasticlogs/tree/main
Thank you in advance!
couldnt make it work, but i will put a link here (its a rar file with the logs): Gofile - Your all-in-one storage solution
If you see everything as one line with default notepad, in notepad++ I see it regularly (one log per line)
The gap in the jstack logs is the ones we need - it looks like Elasticsearch also just froze for tens+ of seconds at that time. I think there's something wrong in your system outside of ES here - I have no other way to explain why jstack and ES might both just hang for that long.
I'm sure its because of the high cpu, but I dont know how to diagnose it. If I will try to catch jstack logs more often (lets say every 5s), maybe it will give us better info?
I don't think it's Elasticsearch, ES itself won't be able to block the entire system like this. I suspect the high CPU is a symptom of something else getting stuck.
I ran "top -b -n 1 -c -H" every 10Sec and this is the first result after it got stuck (and i saw it even before in previous cases that elasticsearch is the one that consuming cpu with htop):
20240116-112256
top - 11:22:57 up 369 days, 20:52, 3 users, load average: 36.34, 31.48, 30.92
Threads: 6661 total, 15 running, 6646 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.6 us, 1.6 sy, 0.0 ni, 76.6 id, 0.1 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 79100940+total, 10132932 free, 32252688+used, 45834963+buff/cache
KiB Swap: 0 total, 0 free, 0 used. 46524675+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
129219 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:20.12 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129407 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.42 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129408 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.76 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129398 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.08 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129399 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.44 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129400 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129401 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.84 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129402 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.45 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129403 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.20 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129404 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:17.36 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129405 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:18.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129406 elastic+ 20 0 1819.5g 41.6g 5.8g R 99.9 5.5 98:19.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128633 elastic+ 20 0 1379.8g 45.5g 9.2g S 60.7 6.0 141:33.08 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
166722 elastic+ 20 0 1999.5g 47.4g 8.7g R 57.1 6.3 2:15.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167736 root 20 0 168936 9052 1616 R 50.0 0.0 0:00.26 top -b -n 1 -c -H
167608 elastic+ 20 0 1819.5g 41.6g 5.8g S 46.4 5.5 0:16.58 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129030 elastic+ 20 0 1379.8g 45.5g 9.2g R 35.7 6.0 21:56.54 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
138637 elastic+ 20 0 1999.5g 47.4g 8.7g S 21.4 6.3 897:46.60 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141011 elastic+ 20 0 1999.5g 47.4g 8.7g S 21.4 6.3 51:43.26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141004 elastic+ 20 0 1999.5g 47.4g 8.7g S 17.9 6.3 55:37.32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141912 elastic+ 20 0 1999.5g 47.4g 8.7g S 14.3 6.3 282:18.38 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143046 elastic+ 20 0 1999.5g 47.4g 8.7g S 14.3 6.3 280:08.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147525 elastic+ 20 0 1700.6g 43.2g 5.1g S 10.7 5.7 141:24.04 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142866 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 284:58.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142921 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 191:43.88 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143223 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 281:28.91 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143233 elastic+ 20 0 1999.5g 47.4g 8.7g S 10.7 6.3 281:18.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129745 elastic+ 20 0 1819.5g 41.6g 5.8g S 7.1 5.5 8:00.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130669 elastic+ 20 0 1819.5g 41.6g 5.8g S 7.1 5.5 33:54.96 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
137756 elastic+ 20 0 1700.6g 43.2g 5.1g S 7.1 5.7 590:22.48 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142949 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 281:56.28 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143247 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 281:11.15 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143287 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 283:22.36 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143283 elastic+ 20 0 1999.5g 47.4g 8.7g S 7.1 6.3 0:09.26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
166654 elastic+ 20 0 1379.8g 45.5g 9.2g S 3.6 6.0 0:32.56 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130628 elastic+ 20 0 1819.5g 41.6g 5.8g S 3.6 5.5 33:56.09 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147527 elastic+ 20 0 1700.6g 43.2g 5.1g S 3.6 5.7 138:26.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141009 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 41:24.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
141080 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 48:11.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142977 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 281:06.01 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147037 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 189:04.43 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
92453 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 2:18.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
122450 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 1:00.61 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131861 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.72 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131870 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.68 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131872 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:18.98 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131873 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:20.21 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131875 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:19.21 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131881 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:19.33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
133167 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:15.76 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
144168 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:08.32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
144584 elastic+ 20 0 1999.5g 47.4g 8.7g S 3.6 6.3 0:09.87 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
147680 elastic+ 20 0 2042.5g 44.8g 6.9g S 3.6 5.9 178:21.45 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
In addition, I tried to put the jstack logs that I sent here and I took the first log after it got stuck (the log from the followers check), and put it in a website that analyze it, and got this:
And also this:
Do you think that it might be right or it just follows simple thumb rules to analyze it?
Can you also share the preceding top output?
Not really, all this waiting in TranslogWriter#add again suggests some kind of blockage outside of ES.
joean407
January 17, 2024, 10:16am
17
Just curious here, do you have 8 disks with different controller/busses on each server or are all disks on the same storage controller?
Sure! These are the top results from the preceding "top -b -n 1 -c -H"
20240116-112200
top - 11:22:46 up 369 days, 20:52, 3 users, load average: 42.13, 32.40, 31.20
Threads: 6665 total, 18 running, 6647 sleeping, 0 stopped, 0 zombie
%Cpu(s): 25.8 us, 26.1 sy, 0.0 ni, 47.8 id, 0.2 wa, 0.0 hi, 0.2 si, 0.0 st
KiB Mem : 79100940+total, 10968244 free, 32263014+used, 45741104+buff/cache
KiB Swap: 0 total, 0 free, 0 used. 46515104+avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
129853 elastic+ 20 0 1819.5g 41.6g 5.8g S 89.5 5.5 146:04.58 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128633 elastic+ 20 0 2007.4g 47.1g 10.8g R 87.5 6.2 141:27.85 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143252 elastic+ 20 0 2007.4g 47.1g 10.8g S 81.5 6.2 25:57.57 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
136366 elastic+ 20 0 2007.4g 47.1g 10.8g S 81.2 6.2 122:41.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
35048 root 20 0 21844 1340 812 S 80.3 0.0 3826:41 /usr/sbin/irqbalance --foreground
129804 elastic+ 20 0 1775.2g 41.1g 5.2g S 79.9 5.5 145:25.33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128630 elastic+ 20 0 2007.4g 47.1g 10.8g S 79.5 6.2 10:21.65 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139494 elastic+ 20 0 1819.5g 41.6g 5.8g S 76.7 5.5 144:13.39 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167608 elastic+ 20 0 1819.5g 41.6g 5.8g S 23.0 5.5 0:12.62 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139660 elastic+ 20 0 1819.5g 41.6g 5.8g S 17.7 5.5 24:47.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128974 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.6 5.5 121:53.25 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128966 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:52.93 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128968 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:53.20 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128971 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:52.41 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128975 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:53.94 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128814 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:49.62 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128967 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:50.50 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128969 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:52.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128970 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:52.18 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128973 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.5 5.5 121:53.73 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128972 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.4 5.5 121:53.54 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128976 elastic+ 20 0 1775.2g 41.1g 5.2g S 17.4 5.5 121:52.18 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129298 elastic+ 20 0 2007.4g 47.1g 10.8g S 13.2 6.2 119:16.82 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129852 elastic+ 20 0 1819.5g 41.6g 5.8g S 12.0 5.5 142:10.29 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167695 elastic+ 20 0 2007.4g 47.1g 10.8g R 10.0 6.2 0:02.02 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130333 elastic+ 20 0 1775.2g 41.1g 5.2g S 9.8 5.5 5:30.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167697 elastic+ 20 0 2007.4g 47.1g 10.8g R 8.8 6.2 0:01.77 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139308 elastic+ 20 0 1775.2g 41.1g 5.2g S 8.7 5.5 24:41.98 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143239 elastic+ 20 0 2007.4g 47.1g 10.8g R 8.2 6.2 24:19.52 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
391 root 20 0 0 0 0 S 7.9 0.0 5998:18 [kswapd0]
140916 elastic+ 20 0 1999.5g 47.4g 8.8g R 5.6 6.3 136:35.13 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
142422 elastic+ 20 0 1700.6g 43.2g 5.2g R 5.6 5.7 263:18.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167691 elastic+ 20 0 1999.5g 47.4g 8.8g R 5.5 6.3 0:01.70 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139486 elastic+ 20 0 1999.5g 47.4g 8.8g S 5.0 6.3 1071:35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139479 elastic+ 20 0 1999.5g 47.4g 8.8g S 5.0 6.3 1071:30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139484 elastic+ 20 0 1999.5g 47.4g 8.8g S 5.0 6.3 1071:25 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139487 elastic+ 20 0 1999.5g 47.4g 8.8g S 5.0 6.3 1071:33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139480 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:29 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139482 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139483 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139485 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:27 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
138464 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139478 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:22 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139481 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.9 6.3 1071:33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139477 elastic+ 20 0 1999.5g 47.4g 8.8g S 4.8 6.3 1071:21 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
392 root 20 0 0 0 0 S 4.5 0.0 2331:59 [kswapd1]
129314 elastic+ 20 0 1819.5g 41.6g 5.8g S 4.5 5.5 105:18.81 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130260 elastic+ 20 0 1775.2g 41.1g 5.2g S 4.5 5.5 5:15.96 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130528 elastic+ 20 0 1775.2g 41.1g 5.2g S 4.4 5.5 6:08.94 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
198894 elastic+ 20 0 2007.4g 47.1g 10.8g S 4.3 6.2 16:04.37 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
146798 elastic+ 20 0 1999.5g 47.4g 8.8g R 4.1 6.3 193:25.14 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130652 elastic+ 20 0 1775.2g 41.1g 5.2g S 4.0 5.5 6:10.50 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130939 elastic+ 20 0 1819.5g 41.6g 5.8g S 4.0 5.5 4:48.53 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
166722 elastic+ 20 0 1999.5g 47.4g 8.8g R 4.0 6.3 2:08.88 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167702 elastic+ 20 0 1689.1g 42.8g 5.4g R 4.0 5.7 0:00.80 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130345 elastic+ 20 0 1775.2g 41.1g 5.2g S 3.6 5.5 5:26.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130265 elastic+ 20 0 1775.2g 41.1g 5.2g S 3.2 5.5 5:49.57 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131285 elastic+ 20 0 1819.5g 41.6g 5.8g S 3.1 5.5 5:41.97 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131829 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.8 5.5 4:34.71 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131167 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.8 5.5 5:05.40 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130571 elastic+ 20 0 1775.2g 41.1g 5.2g S 2.7 5.5 5:57.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
251 root 20 0 0 0 0 S 2.5 0.0 292:23.75 [ksoftirqd/48]
196025 elastic+ 20 0 2007.4g 47.1g 10.8g S 2.5 6.2 9:37.89 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130579 elastic+ 20 0 1775.2g 41.1g 5.2g S 2.5 5.5 6:21.05 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130757 elastic+ 20 0 1775.2g 41.1g 5.2g S 2.4 5.5 5:38.72 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131345 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.3 5.5 4:47.31 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128877 elastic+ 20 0 1775.2g 41.1g 5.2g S 2.3 5.5 113:06.93 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
146019 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.2 5.5 26:41.52 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
128639 elastic+ 20 0 2007.4g 47.1g 10.8g S 2.2 6.2 10:11.90 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129617 elastic+ 20 0 1775.2g 41.1g 5.2g S 2.1 5.5 141:09.41 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131189 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.1 5.5 4:42.23 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131527 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.1 5.5 4:46.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131930 elastic+ 20 0 1819.5g 41.6g 5.8g S 2.0 5.5 4:47.47 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130508 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.9 5.5 5:30.70 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131074 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.9 5.5 5:57.41 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
138637 elastic+ 20 0 1999.5g 47.4g 8.8g R 1.9 6.3 897:45.58 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130933 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.9 5.5 5:06.41 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
138383 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.8 6.2 19:22.57 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130480 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.8 5.5 5:21.63 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130354 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.7 5.5 5:13.62 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130355 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.7 5.5 4:49.97 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131958 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.7 5.5 5:30.48 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139411 elastic+ 20 0 1759.4g 43.7g 6.1g S 1.7 5.8 703:44.40 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130515 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.7 5.5 6:01.91 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130522 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.7 5.5 5:38.27 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131931 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.7 5.5 4:56.31 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129873 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.6 6.2 34:10.81 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131927 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.6 5.5 4:44.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
135126 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.5 6.2 6:16.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
136215 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.5 6.2 5:53.38 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130514 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.5 5.5 5:38.75 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130624 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.5 5.5 34:43.69 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131701 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.5 6.2 34:40.59 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
132056 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.4 6.2 34:10.52 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
167704 elastic+ 20 0 2007.4g 47.1g 10.8g R 1.4 6.2 0:00.28 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
35450 root 0 -20 0 0 0 S 1.3 0.0 84:24.28 [kworker/24:1H]
131247 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.3 6.2 34:11.59 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130189 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.3 5.5 24:11.48 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131879 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.2 5.5 5:49.49 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129026 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.2 6.2 1:56.23 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
196027 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.1 6.2 10:31.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131007 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.1 5.5 4:27.91 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131655 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.1 5.5 6:19.44 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131916 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.1 5.5 4:52.70 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129089 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.1 6.2 19:20.59 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139863 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.1 6.2 24:45.55 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130239 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.1 5.5 5:17.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130408 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.1 5.5 5:45.11 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130812 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.1 5.5 1:47.30 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
139264 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.1 5.5 21:59.76 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131728 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 34:05.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
132075 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 34:27.39 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
16255 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 15:00.38 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130482 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.0 5.5 6:23.96 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130774 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.0 5.5 5:40.32 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130240 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 151:37.71 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131130 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 5:45.66 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131281 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 6:24.04 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131431 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 5:02.28 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
143247 elastic+ 20 0 1999.5g 47.4g 8.8g S 1.0 6.3 281:11.07 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129088 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 12:21.33 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129090 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 5:58.43 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130765 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 34:12.51 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
134927 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 1:07.94 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
136532 elastic+ 20 0 2007.4g 47.1g 10.8g S 1.0 6.2 7:02.95 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130389 elastic+ 20 0 1775.2g 41.1g 5.2g S 1.0 5.5 5:59.39 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131537 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 5:28.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131650 elastic+ 20 0 1819.5g 41.6g 5.8g S 1.0 5.5 5:47.65 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130348 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 35:40.82 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131274 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:37.07 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131756 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:08.80 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131981 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:09.28 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131996 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:26.40 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130247 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.9 5.5 5:46.55 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131319 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.9 5.5 5:25.26 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131868 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.9 5.5 4:36.31 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131191 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:17.52 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131795 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 34:06.69 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
134863 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.9 6.2 5:34.80 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130479 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.9 5.5 6:07.38 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130560 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:02.93 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130647 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:24.67 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131932 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 35:45.45 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130327 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 5:09.17 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130756 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 5:56.89 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130663 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.8 5.5 33:56.56 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129160 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 3:46.25 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
129704 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:46.12 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130936 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:05.77 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131678 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:08.35 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131836 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 33:53.97 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131910 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:35.42 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
132099 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 33:48.40 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
132179 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:18.85 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
132258 elastic+ 20 0 2007.4g 47.1g 10.8g S 0.8 6.2 34:41.79 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130302 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 5:16.70 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130501 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 5:06.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130543 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 5:19.96 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130854 elastic+ 20 0 1775.2g 41.1g 5.2g S 0.8 5.5 4:52.83 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130641 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.8 5.5 33:51.11 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
130855 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.8 5.5 5:41.23 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
131848 elastic+ 20 0 1819.5g 41.6g 5.8g S 0.8 5.5 6:02.42 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -Djava.security.manager=a+
Every server has its own storage controller
elasticexpert:
20240116-112200
top - 11:22:46 up 369 days, 20:52, 3 users, load average: 42.13, 32.40, 31.20
Ok the difference between 112200 and 11:22:46 indicates that it took top 46 seconds to respond right? Elasticsearch is just a regular userspace process, it can't block the whole system like that.
1 Like
Make sense... The servers run only elasticsearch, so maybe from your experience what could cause it? (network? io?)
leandrojmp
January 18, 2024, 2:35pm
24
load average: 42.13, 32.40, 31.20
What are the specs for this server? How many CPUs and RAM does your server have?
Do you have any custom or non-standard configuration in place or are you letting Elasticsearch set settings, e.g. thread pool sizes, by default? Based on the output I asume the nodes are running directly on the host and not in containers. Is that correct?
I haven't seen much like these symptoms before tbh. It sounds to me like something is badly wrong with thread scheduling in the kernel. Possibly some unusual/buggy driver or firmware? Are you using any virtualization here? Maybe a hypervisor problem? Or perhaps you've allocated more vCPUs than you have physical CPUs?
1 Like
The server type is Cisco ucs c240 m5.
