Es crashes with "Native controller process has stopped - no new native processes can be started"

Hello everyone,
when I run the ./bin/elasticsearch command I get the following logs:

[...........]

[2018-08-10T08:14:57,579][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [aggs-matrix-stats]
[2018-08-10T08:14:57,579][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [analysis-common]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [ingest-common]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [lang-expression]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [lang-mustache]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [lang-painless]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [mapper-extras]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [parent-join]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [percolator]
[2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [rank-eval]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [reindex]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [repository-url]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [transport-netty4]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [tribe]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-core]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-deprecation]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-graph]
[2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-logstash]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-ml]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-monitoring]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-rollup]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-security]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-sql]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-upgrade]
[2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] loaded module [x-pack-watcher]
[2018-08-10T08:14:57,583][INFO ][o.e.p.PluginsService ] [ZGgC7Hb] no plugins loaded
[2018-08-10T08:15:02,490][INFO ][o.e.x.s.a.s.FileRolesStore] [ZGgC7Hb] parsed [0] roles from file [/home/andrea2/elasticsearch-6.3.2/config/roles.yml]
[2018-08-10T08:15:03,771][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/2436] [Main.cc@109] controller (64 bit): Version 6.3.2 (Build 903094f295d249) Copyright (c) 2018 Elasticsearch BV
[2018-08-10T08:15:04,352][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2018-08-10T08:15:04,694][INFO ][o.e.d.DiscoveryModule ] [ZGgC7Hb] using discovery type [zen]
[2018-08-10T08:15:05,915][INFO ][o.e.n.Node ] [ZGgC7Hb] initialized
[2018-08-10T08:15:05,915][INFO ][o.e.n.Node ] [ZGgC7Hb] starting ...
[2018-08-10T08:15:06,130][INFO ][o.e.t.TransportService ] [ZGgC7Hb] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2018-08-10T08:15:09,241][INFO ][o.e.c.s.MasterService ] [ZGgC7Hb] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[2018-08-10T08:15:09,247][INFO ][o.e.c.s.ClusterApplierService] [ZGgC7Hb] new_master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
[2018-08-10T08:15:09,293][INFO ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [ZGgC7Hb] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2018-08-10T08:15:09,294][INFO ][o.e.n.Node ] [ZGgC7Hb] started
[2018-08-10T08:15:09,702][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [ZGgC7Hb] Failed to clear cache for realms [[]]
[2018-08-10T08:15:09,732][INFO ][o.e.l.LicenseService ] [ZGgC7Hb] license [e4dc900d-e45c-4547-a2c8-32a454aa57bb] mode [basic] - valid
[2018-08-10T08:15:09,750][INFO ][o.e.g.GatewayService ] [ZGgC7Hb] recovered [0] indices into cluster_state
[2018-08-10T08:15:54,799][INFO ][o.e.x.m.j.p.NativeController] Native controller process has stopped - no new native processes can be started
[2018-08-10T08:15:54,801][INFO ][o.e.n.Node ] [ZGgC7Hb] stopping ...
[2018-08-10T08:15:54,812][INFO ][o.e.x.w.WatcherService ] [ZGgC7Hb] stopping watch service, reason [shutdown initiated]
[2018-08-10T08:15:55,259][INFO ][o.e.n.Node ] [ZGgC7Hb] stopped
[2018-08-10T08:15:55,259][INFO ][o.e.n.Node ] [ZGgC7Hb] closing ...
[2018-08-10T08:15:55,272][INFO ][o.e.n.Node ] [ZGgC7Hb] closed

Why is it not closed?

Thanks in advance

Hi Andrea,

What operating system are you running on?

Also, is this a clean install of elasticsearch? Or have you upgraded an existing one?

Kind regards,
Dimitris

Also, there's a large gap in the logs between 08:15:09 and 08:15:54. If you get the same problem every time you start the node, please could you try running with DEBUG level logging so that we can see if any additional messages get logged during the gap. It seems that something is timing out causing the node to shut itself down, but it's not completely clear what.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.