# Server disk broken and recover, but failed to start **URL:** \ **Category:** Elasticsearch\ **Created:** [July 22, 2020, 6:51am UTC](https://discuss.elastic.co/t/server-disk-broken-and-recover-but-failed-to-start/242140 "2020-07-22T06:51:18Z")\ **Posts on this page:** 1\ **Showing post:** 6
**Author:** ![Steve\_Mushero](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steve_mushero/32/22441_2.png) [@Steve\_Mushero](https://discuss.elastic.co/u/Steve_Mushero)\ **Post date:** [July 23, 2020, 4:44am UTC](https://discuss.elastic.co/t/server-disk-broken-and-recover-but-failed-to-start/242140/6 "2020-07-23T04:44:21Z")
Why is -Xms, -Xmx on the command line twice? Both for 20G and 4G? This in docker, as I see OsProbe.getCgroup as last area the exception happened, which is odd - any other strange settings as it's still probing for OS and related settings. Looks like a CentOS VM but maybe some weird cgroup settings, like cgroup is returning null, which I suspect it should not do (I think Linux has a default cgroup). Seems this happens sometimes, see these and fixed in V7.4 > > > \`\`\` > \]\[WARN \]\[o.e.b.ElasticsearchUncaughtExceptionHandler\] \[\*\*\*\*\*\*\*\*\] uncaught e…xception in thread \[main\] > org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.cli.Command.main(Command.java:90) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~\[elasticsearch-5.6.3.jar:5.6.3\] > Caused by: java.lang.NullPointerException > at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~\[?:?\] > at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:274) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:261) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:419) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:464) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.os.OsService.\(OsService.java:45) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.monitor.MonitorService.\(MonitorService.java:45) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.node.Node.\(Node.java:362) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.node.Node.\(Node.java:245) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Bootstrap$5.\(Bootstrap.java:233) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~\[elasticsearch-5.6.3.jar:5.6.3\] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~\[elasticsearch-5.6.3.jar:5.6.3\] > \`\`\` > > > \*\*Elasticsearch version\*\* (\`bin/elasticsearch --version\`): > 6.8.1 (Gentoo ebuild…) > running in a systemd nspawn container > > \*\*Plugins installed\*\*: \[\] > > \*\*JVM version\*\* (\`java -version\`): > openjdk version "1.8.0\_212" > OpenJDK Runtime Environment (IcedTea 3.12.0) (Gentoo icedtea-3.12.0) > OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) > > \*\*OS version\*\* (\`uname -a\` if on a Unix-like system): > Linux es6 5.2.7-gentoo #1 SMP Fri Aug 9 11:40:38 CEST 2019 x86\_64 Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz GenuineIntel GNU/Linux > > \*\*Description of the problem including expected versus actual behavior\*\*: > Since upgrading ES in my development container, it no longer starts. It crashes with a stack trace pointing to problems determining cgroup controllers. Researching other reports showed that the problem should be fixed, but it still shows for me. Logs below. > > Last known good version: > \`Sat Sep 15 02:21:22 2018 \>\>\> app-misc/elasticsearch-6.4.0\` > > \*\*Steps to reproduce\*\*: > > \`\`\` > es6 ~ # ls -l /sys/fs/cgroup > insgesamt 0 > drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 blkio > drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 cpu > lrwxrwxrwx 1 root root 3 9. Aug 17:42 cpuacct -\> cpu > drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 devices > drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 memory > dr-xr-xr-x 2 nobody nobody 0 9. Aug 11:57 net\_cls > lrwxrwxrwx 1 root root 7 9. Aug 17:42 net\_prio -\> net\_cls > drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 pids > drwxr-xr-x 5 root root 0 9. Aug 17:42 systemd > drwxr-xr-x 5 root root 0 9. Aug 17:42 unified > > es6 ~ # cat /proc/cgroups > \#subsys\_name hierarchy num\_cgroups enabled > cpu 2 88 1 > blkio 4 108 1 > memory 5 128 1 > devices 3 86 1 > net\_cls 6 1 1 > pids 7 92 1 > > es6 ~ # cat /proc/self/cgroup > 7:pids:/ > 6:net\_cls:/ > 5:memory:/ > 4:blkio:/ > 3:devices:/ > 2:cpu:/ > 1:name=systemd:/user.slice/user-0.slice/session-c1.scope > 0::/user.slice/user-0.slice/session-c1.scope > > es6 ~ # mount -t cgroup > cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) > cgroup on /sys/fs/cgroup/net\_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net\_cls) > cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) > cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) > cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu) > cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) > cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) > > es6 ~ # grep cgroup /proc/self/mountinfo > 1067 1141 0:106 / /sys/fs/cgroup ro,nosuid,nodev,noexec - tmpfs tmpfs ro,mode=755,uid=65536,gid=65536 > 1068 1067 0:29 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory > 1069 1067 0:30 / /sys/fs/cgroup/net\_cls rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net\_cls > 1070 1067 0:27 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices > 1071 1067 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio > 1073 1067 0:26 / /sys/fs/cgroup/cpu rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpu > 1074 1067 0:31 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids > 1075 1067 0:21 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate > 1076 1067 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,xattr,name=systemd > > \# FULL LOG AT THE END > es6 ~ # systemctl start elasticsearch.service > es6 ~ # systemctl status elasticsearch.service > ● elasticsearch.service - Elasticsearch > Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) > Drop-In: /etc/systemd/system/elasticsearch.service.d > └─00gentoo.conf, override.conf > Active: failed (Result: exit-code) since Fri 2019-08-09 18:18:28 CEST; 640ms ago > Docs: https://www.elastic.co > Process: 437 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS) > Process: 438 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID\_DIR}/elasticsearch.pid -Epath.logs=${LOG\_DIR} -Epath.data=${DATA\_DIR} (code=exited, status=1/FAILURE) > Main PID: 438 (code=exited, status=1/FAILURE) > > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.monitor.MonitorService.\(MonitorService.java:46) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.node.Node.\(Node.java:399) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.node.Node.\(Node.java:266) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.bootstrap.Bootstrap$5.\(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Aug 09 18:18:28 es6 elasticsearch\[438\]: ... 6 more > Aug 09 18:18:28 es6 systemd\[1\]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE > Aug 09 18:18:28 es6 systemd\[1\]: elasticsearch.service: Failed with result 'exit-code'. > \`\`\` > > \*\*Provide logs (if relevant)\*\*: > \`\`\` > java.lang.NullPointerException: null > at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~\[?:?\] > at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~\[elasticsearch-core-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:277) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:264) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:483) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:603) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsService.\(OsService.java:49) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.MonitorService.\(MonitorService.java:46) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.node.Node.\(Node.java:399) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.node.Node.\(Node.java:266) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap$5.\(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) \[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) \[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) \[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) \[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) \[elasticsearch-cli-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.Command.main(Command.java:90) \[elasticsearch-cli-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) \[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) \[elasticsearch-6.8.1.jar:6.8.1\] > \[2019-08-09T17:46:33,105\]\[WARN \]\[o.e.b.ElasticsearchUncaughtExceptionHandler\] \[QoY4z88\] uncaught exception in thread \[main\] > org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~\[elasticsearch-cli-6.8.1.jar:6.8.1\] > at org.elasticsearch.cli.Command.main(Command.java:90) ~\[elasticsearch-cli-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~\[elasticsearch-6.8.1.jar:6.8.1\] > Caused by: java.lang.NullPointerException > at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~\[?:?\] > at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~\[elasticsearch-core-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:277) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:264) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:483) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:603) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.os.OsService.\(OsService.java:49) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.monitor.MonitorService.\(MonitorService.java:46) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.node.Node.\(Node.java:399) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.node.Node.\(Node.java:266) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap$5.\(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~\[elasticsearch-6.8.1.jar:6.8.1\] > at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~\[elasticsearch-6.8.1.jar:6.8.1\] > ... 6 more > \`\`\` Also some workarounds here: > [@Why elasticsearch-5.2.0 throw NullPointerException when start](https://discuss.elastic.co/t/why-elasticsearch-5-2-0-throw-nullpointerexception-when-start/74739/14): > > Sorry, I don't know. Okay, you can work around this with: $ sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpu,cpuacct cgroup /sys/fs/cgroup/cpu,cpuacct --- _[View the full topic](https://discuss.elastic.co/t/server-disk-broken-and-recover-but-failed-to-start/242140)._