Hello Support,
We are currently running a es 2.3.3 cluster and trying to upgrade 6.x.
While running the _reindex job we are running into this issue.
Native controller process has stopped - no new native processes can be started. Logs are attached.
Thanks,
Ashok
`Java HotSpot(TM) 64-Bit Server VM (25.131-b11) for linux-amd64 JRE (1.8.0_131-b11), built on Mar 15 2017 01:23:40 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)
Memory: 4k page, physical 16265876k(15011776k free), swap 0k(0k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:InitialHeapSize=10737418240 -XX:MaxHeapSize=10737418240 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:NewSize=348966912 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=697933824 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2019-01-25T16:58:05.095+0000: 64595.302: Total time for which application threads were stopped: 0.0004850 seconds, Stopping threads took: 0.0000752 seconds
2019-01-25T16:58:06.936+0000: 64597.143: [GC (Allocation Failure) 2019-01-25T16:58:06.936+0000: 64597.143: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6)
- age 1: 22086200 bytes, 22086200 total
- age 2: 175744 bytes, 22261944 total
: 300193K->24094K(306688K), 0.0397693 secs] 3349497K->3073398K(10451712K), 0.0398951 secs] [Times: user=0.15 sys=0.00, real=0.04 secs]
2019-01-25T16:58:06.976+0000: 64597.183: Total time for which application threads were stopped: 0.0405179 seconds, Stopping threads took: 0.0000714 seconds
2019-01-25T16:58:08.977+0000: 64599.183: Total time for which application threads were stopped: 0.0004849 seconds, Stopping threads took: 0.0000709 seconds
2019-01-25T16:58:11.426+0000: 64601.632: [GC (Allocation Failure) 2019-01-25T16:58:11.426+0000: 64601.632: [ParNew
Desired survivor size 17432576 bytes, new threshold 6 (max 6) - age 1: 13319832 bytes, 13319832 total
: 296734K->27297K(306688K), 0.0324607 secs] 3346038K->3094781K(10451712K), 0.0325800 secs] [Times: user=0.12 sys=0.01, real=0.04 secs]
2019-01-25T16:58:11.458+0000: 64601.665: Total time for which application threads were stopped: 0.0332405 seconds, Stopping threads took: 0.0001078 seconds
2019-01-25T16:58:11.672+0000: 64601.878: [GC (Allocation Failure) 2019-01-25T16:58:11.672+0000: 64601.878: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6) - age 1: 30664120 bytes, 30664120 total
- age 2: 4190968 bytes, 34855088 total
: 299937K->34048K(306688K), 0.0346497 secs] 3367421K->3124483K(10451712K), 0.0347958 secs] [Times: user=0.13 sys=0.01, real=0.03 secs]
2019-01-25T16:58:11.707+0000: 64601.913: Total time for which application threads were stopped: 0.0356858 seconds, Stopping threads took: 0.0001638 seconds
2019-01-25T16:58:11.890+0000: 64602.097: [GC (Allocation Failure) 2019-01-25T16:58:11.890+0000: 64602.097: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6) - age 1: 18607000 bytes, 18607000 total
: 306688K->27281K(306688K), 0.0348845 secs] 3397123K->3135339K(10451712K), 0.0349948 secs] [Times: user=0.13 sys=0.00, real=0.03 secs]
2019-01-25T16:58:11.925+0000: 64602.132: Total time for which application threads were stopped: 0.0356127 seconds, Stopping threads took: 0.0000761 seconds
2019-01-25T16:58:12.483+0000: 64602.689: [GC (Allocation Failure) 2019-01-25T16:58:12.483+0000: 64602.689: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6) - age 1: 34835632 bytes, 34835632 total
: 299921K->34048K(306688K), 0.0589977 secs] 3407979K->3201067K(10451712K), 0.0591195 secs] [Times: user=0.22 sys=0.01, real=0.06 secs]
2019-01-25T16:58:12.542+0000: 64602.748: Total time for which application threads were stopped: 0.0604470 seconds, Stopping threads took: 0.0007300 seconds
2019-01-25T16:58:13.702+0000: 64603.908: [GC (Allocation Failure) 2019-01-25T16:58:13.702+0000: 64603.908: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6) - age 1: 21085456 bytes, 21085456 total
: 306688K->34048K(306688K), 0.0415446 secs] 3611604K->3402773K(10451712K), 0.0416910 secs] [Times: user=0.14 sys=0.01, real=0.04 secs]
2019-01-25T16:58:13.744+0000: 64603.950: Total time for which application threads were stopped: 0.0424597 seconds, Stopping threads took: 0.0001580 seconds
2019-01-25T16:58:14.744+0000: 64604.951: Total time for which application threads were stopped: 0.0006177 seconds, Stopping threads took: 0.0001557 seconds
2019-01-25T16:58:15.075+0000: 64605.282: [GC (Allocation Failure) 2019-01-25T16:58:15.075+0000: 64605.282: [ParNew
Desired survivor size 17432576 bytes, new threshold 6 (max 6) - age 1: 8669968 bytes, 8669968 total
: 306688K->22913K(306688K), 0.0366591 secs] 3675413K->3410750K(10451712K), 0.0367812 secs] [Times: user=0.14 sys=0.00, real=0.04 secs]
2019-01-25T16:58:15.112+0000: 64605.319: Total time for which application threads were stopped: 0.0374522 seconds, Stopping threads took: 0.0000585 seconds
2019-01-25T16:58:15.591+0000: 64605.797: Total time for which application threads were stopped: 0.0006085 seconds, Stopping threads took: 0.0000910 seconds
2019-01-25T16:58:17.344+0000: 64607.551: [GC (Allocation Failure) 2019-01-25T16:58:17.344+0000: 64607.551: [ParNew
Desired survivor size 17432576 bytes, new threshold 1 (max 6) - age 1: 30540240 bytes, 30540240 total
- age 2: 4264304 bytes, 34804544 total
: 295553K->34048K(306688K), 0.0458444 secs] 3683390K->3424984K(10451712K), 0.0459782 secs] [Times: user=0.18 sys=0.01, real=0.05 secs]
2019-01-25T16:58:17.390+0000: 64607.597: Total time for which application threads were stopped: 0.0466295 seconds, Stopping threads took: 0.0000759 seconds
2019-01-25T16:58:18.391+0000: 64608.597: Total time for which application threads were stopped: 0.0005970 seconds, Stopping threads took: 0.0001250 seconds
2019-01-25T16:58:18.910+0000: 64609.116: [GC (Allocation Failure) 2019-01-25T16:58:18.910+0000: 64609.116: [ParNew
Desired survivor size 17432576 bytes, new threshold 6 (max 6) - age 1: 4691752 bytes, 4691752 total
: 306215K->27512K(306688K), 0.0321921 secs] 3697151K->3433345K(10451712K), 0.0323056 secs] [Times: user=0.12 sys=0.00, real=0.03 secs]
2019-01-25T16:58:18.942+0000: 64609.149: Total time for which application threads were stopped: 0.0329126 seconds, Stopping threads took: 0.0000629 seconds`