Why elasticsearch-5.2.0 throw NullPointerException when start

Elastic Stack Elasticsearch
1

mr@zdh108:~/elasticsearch-5.2.0/bin> java -version
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
mr@zdh108:~/elasticsearch-5.2.0/bin> cat /etc/issue

Welcome to SUSE Linux Enterprise Server 12 SP1 (x86_64) - Kernel \r (\l).

mr@zdh108:~/elasticsearch-5.2.0/bin> ./elasticsearch
[2017-02-11T11:15:06,585][INFO ][o.e.n.Node ] [] initializing ...
[2017-02-11T11:15:06,690][INFO ][o.e.e.NodeEnvironment ] [KCqSYpD] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [132.4gb], net total_space [179.9gb], spins? [unknown], types [rootfs]
[2017-02-11T11:15:06,690][INFO ][o.e.e.NodeEnvironment ] [KCqSYpD] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-02-11T11:15:06,691][INFO ][o.e.n.Node ] node name [KCqSYpD] derived from node ID [KCqSYpDhTbKEPkHxB1UDxg]; set [node.name] to override
[2017-02-11T11:15:06,694][INFO ][o.e.n.Node ] version[5.2.0], pid[52315], build[24e05b9/2017-01-24T19:52:35.800Z], OS[Linux/3.12.49-11-default/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-02-11T11:15:07,666][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [aggs-matrix-stats]
[2017-02-11T11:15:07,666][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [ingest-common]
[2017-02-11T11:15:07,666][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [lang-expression]
[2017-02-11T11:15:07,666][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [lang-groovy]
[2017-02-11T11:15:07,667][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [lang-mustache]
[2017-02-11T11:15:07,667][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [lang-painless]
[2017-02-11T11:15:07,667][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [percolator]
[2017-02-11T11:15:07,667][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [reindex]
[2017-02-11T11:15:07,667][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [transport-netty3]
[2017-02-11T11:15:07,668][INFO ][o.e.p.PluginsService ] [KCqSYpD] loaded module [transport-netty4]
[2017-02-11T11:15:07,668][INFO ][o.e.p.PluginsService ] [KCqSYpD] no plugins loaded
[2017-02-11T11:15:08,442][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.2.0.jar:5.2.0]
Caused by: java.lang.NullPointerException
at sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:273) ~[?:?]
at org.elasticsearch.common.io.PathUtils.get(PathUtils.java:60) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.readSysFsCgroupCpuAcctCpuAcctUsage(OsProbe.java:274) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroupCpuAcctUsageNanos(OsProbe.java:261) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.getCgroup(OsProbe.java:419) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsProbe.osStats(OsProbe.java:466) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.os.OsService.(OsService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.monitor.MonitorService.(MonitorService.java:45) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.node.Node.(Node.java:345) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.node.Node.(Node.java:232) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Bootstrap$6.(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:241) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.2.0.jar:5.2.0]
... 6 more

2

Okay, so clearly the CPU control group is coming back null, now we need to figure out why. Can you please provide the output of tree -L 2 /sys/fs/cgroup and do the following?

  1. Start Elasticsearch with ES_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" ./elasticsearch
  2. When Elasticsearch starts, it will print Listening for transport dt_socket at address: 5005 to the console. Let the process hang here.
  3. When this message appears, execute ps aux | grep Elasticsearch and obtain the PID for the Elasticsearch process just started.
  4. Then, provide the output of cat /proc/<PID>/cgroup (with <PID> replaced by the PID obtained in step 3).
  5. Terminate the Elasticsearch process.

Thanks.

3

zdh108:~ # tree -L 2 /sys/fs/cgroup
/sys/fs/cgroup
├── blkio
│ ├── blkio.io_merged
│ ├── blkio.io_merged_recursive
│ ├── blkio.io_queued
│ ├── blkio.io_queued_recursive
│ ├── blkio.io_service_bytes
│ ├── blkio.io_service_bytes_recursive
│ ├── blkio.io_service_time
│ ├── blkio.io_service_time_recursive
│ ├── blkio.io_serviced
│ ├── blkio.io_serviced_recursive
│ ├── blkio.io_wait_time
│ ├── blkio.io_wait_time_recursive
│ ├── blkio.leaf_weight
│ ├── blkio.leaf_weight_device
│ ├── blkio.reset_stats
│ ├── blkio.sectors
│ ├── blkio.sectors_recursive
│ ├── blkio.throttle.io_service_bytes
│ ├── blkio.throttle.io_serviced
│ ├── blkio.throttle.read_bps_device
│ ├── blkio.throttle.read_iops_device
│ ├── blkio.throttle.write_bps_device
│ ├── blkio.throttle.write_iops_device
│ ├── blkio.time
│ ├── blkio.time_recursive
│ ├── blkio.weight
│ ├── blkio.weight_device
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── cpu -> cpu,cpuacct
├── cpu,cpuacct
│ └── cgroup.procs
├── cpuacct -> cpu,cpuacct
├── cpuset
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── cpuset.cpu_exclusive
│ ├── cpuset.cpus
│ ├── cpuset.mem_exclusive
│ ├── cpuset.mem_hardwall
│ ├── cpuset.memory_migrate
│ ├── cpuset.memory_pressure
│ ├── cpuset.memory_pressure_enabled
│ ├── cpuset.memory_spread_page
│ ├── cpuset.memory_spread_slab
│ ├── cpuset.mems
│ ├── cpuset.sched_load_balance
│ ├── cpuset.sched_relax_domain_level
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── devices
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── devices.allow
│ ├── devices.deny
│ ├── devices.list
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── freezer
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── hugetlb
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── hugetlb.2MB.failcnt
│ ├── hugetlb.2MB.limit_in_bytes
│ ├── hugetlb.2MB.max_usage_in_bytes
│ ├── hugetlb.2MB.usage_in_bytes
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── memory
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── memory.failcnt
│ ├── memory.force_empty
│ ├── memory.limit_in_bytes
│ ├── memory.low_limit_in_bytes
│ ├── memory.max_usage_in_bytes
│ ├── memory.move_charge_at_immigrate
│ ├── memory.numa_stat
│ ├── memory.oom_control
│ ├── memory.pressure_level
│ ├── memory.soft_limit_in_bytes
│ ├── memory.stat
│ ├── memory.swappiness
│ ├── memory.usage_in_bytes
│ ├── memory.use_hierarchy
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
├── perf_event
│ ├── cgroup.clone_children
│ ├── cgroup.event_control
│ ├── cgroup.procs
│ ├── cgroup.sane_behavior
│ ├── notify_on_release
│ ├── release_agent
│ └── tasks
└── systemd
├── cgroup.clone_children
├── cgroup.event_control
├── cgroup.procs
├── cgroup.sane_behavior
├── notify_on_release
├── release_agent
├── system.slice
├── tasks
└── user.slice

zdh108:~ # ps aux | grep Elasticsearch
mr 89829 4.5 10.5 4801748 2179528 pts/0 Tl 10:21 0:01 /usr/java/jdk/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 -Des.path.home=/home/mr/elasticsearch-5.2.0 -cp /home/mr/elasticsearch-5.2.0/lib/elasticsearch-5.2.0.jar:/home/mr/elasticsearch-5.2.0/lib/* org.elasticsearch.bootstrap.Elasticsearch
root 93813 0.0 0.0 9236 932 pts/3 S+ 10:22 0:00 grep --color=auto Elasticsearch
zdh108:~ # cat /proc/89829/cgroup
14:cpu:/
10:hugetlb:/
9:perf_event:/
8:blkio:/
7:freezer:/
6:devices:/
5:memory:/
3:cpuset:/
2:name=systemd:/system.slice/sshd.service

4

This is really helpful, thank you. Can you also provide the output of mount | grep cgroup?

5

zdh108:~ # mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
none on /home/dapmanager-agent/cgroup/cpu type cgroup (rw,relatime,cpu)

6

Your system is missing cgroupfs mounts with options cpu and cpuacct. This is unexpected; do you know why that is?

7

This is the case when the Linux kernel cpu scheduler is replaced by another scheduler, like BFS for example.

http://ck.kolivas.org/patches/bfs/bfs-faq.txt

BFS does NOT implement CGROUPS.

8

But, in case of SLES 12, I rather suspect the system does not mount the control group subsystems by default.

https://www.suse.com/documentation/sles-12/singlehtml/book_sle_tuning/book_sle_tuning.html#sec.tuning.cgroups.subsys

9

Thanks @jprante, I'm aware of the possibility of BFS leading to this, but that does not seem to be the case here as it appears to be a stock SLES 12 SP1 kernel.

1 Like
10

I don't think this is the case, but I will check in a stock VM.

11

I installed a stock VM from the ISO SLE-12-SP1-Server-DVD-x86_64-GM-DVD1.iso (MD5: 21ca19898f20866272b8656df82a945e):

20:25:08 [jason:~] $ ssh -p 2222 localhost
Password: 
Last login: Fri Mar  3 20:33:32 2017 from localhost
jason@linux-u4wy:~> cat /etc/issue

Welcome to SUSE Linux Enterprise Server 12 SP1  (x86_64) - Kernel \r (\l).


jason@linux-u4wy:~> uname -a
Linux linux-u4wy 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux
jason@linux-u4wy:~> mount | grep cgroup
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpuacct,cpu)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
jason@linux-u4wy:~>
12

Another interesting point here is how /sys/fs/cgroup/cpu and /sys/fs/cgroup/cpuacct exist despite cgroupfs mounts with mount options cpu and cpuacct not existing.

13

Sorry, I don't know.

14

Sorry, I don't know.

Okay, you can work around this with:

$ sudo mount -t cgroup -o rw,nosuid,nodev,noexec,relatime,cpu,cpuacct cgroup /sys/fs/cgroup/cpu,cpuacct
1 Like
15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.