Server disk broken and recover, but failed to start

Steve_Mushero · July 23, 2020, 4:44am

Why is -Xms, -Xmx on the command line twice? Both for 20G and 4G?

This in docker, as I see OsProbe.getCgroup as last area the exception happened, which is odd - any other strange settings as it's still probing for OS and related settings.

Looks like a CentOS VM but maybe some weird cgroup settings, like cgroup is returning null, which I suspect it should not do (I think Linux has a default cgroup).

Seems this happens sometimes, see these and fixed in V7.4

github.com/elastic/elasticsearch

ES not able to start

opened 08:14AM - 17 Oct 17 UTC

closed 02:17AM - 14 Mar 18 UTC

hello4usharath

:Core/Infra/Core

``` ][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [********] uncaught e…xception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.3.jar:5.6.3] Caused by: java.lang.NullPointerException at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~[?:?] at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:274) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:261) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:419) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:464) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:45) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:45) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.<init>(Node.java:362) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.3.jar:5.6.3] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.3.jar:5.6.3] ```

github.com/elastic/elasticsearch

ES 6.8.1 (also 7.2.1) fails to start (probably due to cgroups)

opened 04:20PM - 09 Aug 19 UTC

closed 08:26AM - 16 Aug 19 UTC

kakra

:Core/Infra/Core

**Elasticsearch version** (`bin/elasticsearch --version`): 6.8.1 (Gentoo ebuild…) running in a systemd nspawn container **Plugins installed**: [] **JVM version** (`java -version`): openjdk version "1.8.0_212" OpenJDK Runtime Environment (IcedTea 3.12.0) (Gentoo icedtea-3.12.0) OpenJDK 64-Bit Server VM (build 25.212-b04, mixed mode) **OS version** (`uname -a` if on a Unix-like system): Linux es6 5.2.7-gentoo #1 SMP Fri Aug 9 11:40:38 CEST 2019 x86_64 Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz GenuineIntel GNU/Linux **Description of the problem including expected versus actual behavior**: Since upgrading ES in my development container, it no longer starts. It crashes with a stack trace pointing to problems determining cgroup controllers. Researching other reports showed that the problem should be fixed, but it still shows for me. Logs below. Last known good version: `Sat Sep 15 02:21:22 2018 >>> app-misc/elasticsearch-6.4.0` **Steps to reproduce**: ``` es6 ~ # ls -l /sys/fs/cgroup insgesamt 0 drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 blkio drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 cpu lrwxrwxrwx 1 root root 3 9. Aug 17:42 cpuacct -> cpu drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 devices drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 memory dr-xr-xr-x 2 nobody nobody 0 9. Aug 11:57 net_cls lrwxrwxrwx 1 root root 7 9. Aug 17:42 net_prio -> net_cls drwxr-xr-x 2 nobody nobody 0 9. Aug 17:42 pids drwxr-xr-x 5 root root 0 9. Aug 17:42 systemd drwxr-xr-x 5 root root 0 9. Aug 17:42 unified es6 ~ # cat /proc/cgroups #subsys_name hierarchy num_cgroups enabled cpu 2 88 1 blkio 4 108 1 memory 5 128 1 devices 3 86 1 net_cls 6 1 1 pids 7 92 1 es6 ~ # cat /proc/self/cgroup 7:pids:/ 6:net_cls:/ 5:memory:/ 4:blkio:/ 3:devices:/ 2:cpu:/ 1:name=systemd:/user.slice/user-0.slice/session-c1.scope 0::/user.slice/user-0.slice/session-c1.scope es6 ~ # mount -t cgroup cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls) cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) cgroup on /sys/fs/cgroup/cpu type cgroup (rw,nosuid,nodev,noexec,relatime,cpu) cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) es6 ~ # grep cgroup /proc/self/mountinfo 1067 1141 0:106 / /sys/fs/cgroup ro,nosuid,nodev,noexec - tmpfs tmpfs ro,mode=755,uid=65536,gid=65536 1068 1067 0:29 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,memory 1069 1067 0:30 / /sys/fs/cgroup/net_cls rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,net_cls 1070 1067 0:27 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,devices 1071 1067 0:28 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,blkio 1073 1067 0:26 / /sys/fs/cgroup/cpu rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,cpu 1074 1067 0:31 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,pids 1075 1067 0:21 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,nsdelegate 1076 1067 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime - cgroup cgroup rw,xattr,name=systemd # FULL LOG AT THE END es6 ~ # systemctl start elasticsearch.service es6 ~ # systemctl status elasticsearch.service ● elasticsearch.service - Elasticsearch Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/elasticsearch.service.d └─00gentoo.conf, override.conf Active: failed (Result: exit-code) since Fri 2019-08-09 18:18:28 CEST; 640ms ago Docs: https://www.elastic.co Process: 437 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS) Process: 438 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid -Epath.logs=${LOG_DIR} -Epath.data=${DATA_DIR} (code=exited, status=1/FAILURE) Main PID: 438 (code=exited, status=1/FAILURE) Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:46) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1] Aug 09 18:18:28 es6 elasticsearch[438]: ... 6 more Aug 09 18:18:28 es6 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE Aug 09 18:18:28 es6 systemd[1]: elasticsearch.service: Failed with result 'exit-code'. ``` **Provide logs (if relevant)**: ``` java.lang.NullPointerException: null at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~[?:?] at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~[elasticsearch-core-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:277) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:264) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:483) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:603) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:49) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:46) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1] [2019-08-09T17:46:33,105][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [QoY4z88] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1] at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1] Caused by: java.lang.NullPointerException at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~[?:?] at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~[elasticsearch-core-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:277) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:264) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:483) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:603) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.os.OsService.<init>(OsService.java:49) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.monitor.MonitorService.<init>(MonitorService.java:46) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.node.Node.<init>(Node.java:399) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1] ... 6 more ```

Also some workarounds here: