Es crashes with "Native controller process has stopped - no new native processes can be started"

Andrea_Boccaforno · August 10, 2018, 12:40pm

Hello everyone,
when I run the ./bin/elasticsearch command I get the following logs:

[...........]

[2018-08-10T08:14:57,579][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,579][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,580][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,581][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,582][INFO ][o.e.p.PluginsService [2018-08-10T08:14:57,583][INFO ][o.e.p.PluginsService [2018-08-10T08:15:02,490][INFO ][o.e.x.s.a.s.FileRolesStore] [2018-08-10T08:15:03,771][INFO ][o.e.x.m.j.p.l.CppLog [2018-08-10T08:15:04,352][DEBUG][o.e.a.ActionModule [2018-08-10T08:15:04,694][INFO ][o.e.d.DiscoveryModule [2018-08-10T08:15:05,915][INFO ][o.e.n.Node [2018-08-10T08:15:05,915][INFO ][o.e.n.Node [2018-08-10T08:15:06,130][INFO ][o.e.t.TransportService [2018-08-10T08:15:09,241][INFO ][o.e.c.s.MasterService [2018-08-10T08:15:09,247][INFO ][o.e.c.s.ClusterApplierService] [2018-08-10T08:15:09,293][INFO ][o.e.x.s.t.n.Security [2018-08-10T08:15:09,294][INFO ][o.e.n.Node [2018-08-10T08:15:09,702][WARN ][o.e.x.s.a.s.m.Native [2018-08-10T08:15:09,732][INFO ][o.e.l.LicenseService [2018-08-10T08:15:09,750][INFO ][o.e.g.GatewayService [2018-08-10T08:15:54,799][INFO ][o.e.x.m.j.p.NativeController] [2018-08-10T08:15:54,801][INFO ][o.e.n.Node [2018-08-10T08:15:54,812][INFO ][o.e.x.w.WatcherService [2018-08-10T08:15:55,259][INFO ][o.e.n.Node [2018-08-10T08:15:55,259][INFO ][o.e.n.Node [2018-08-10T08:15:55,272][INFO ][o.e.n.Node ] [ZGgC7Hb] loaded module [aggs-matrix-stats]
] [ZGgC7Hb] loaded module [analysis-common]
] [ZGgC7Hb] loaded module [ingest-common]
] [ZGgC7Hb] loaded module [lang-expression]
] [ZGgC7Hb] loaded module [lang-mustache]
] [ZGgC7Hb] loaded module [lang-painless]
] [ZGgC7Hb] loaded module [mapper-extras]
] [ZGgC7Hb] loaded module [parent-join]
] [ZGgC7Hb] loaded module [percolator]
] [ZGgC7Hb] loaded module [rank-eval]
] [ZGgC7Hb] loaded module [reindex]
] [ZGgC7Hb] loaded module [repository-url]
] [ZGgC7Hb] loaded module [transport-netty4]
] [ZGgC7Hb] loaded module [tribe]
] [ZGgC7Hb] loaded module [x-pack-core]
] [ZGgC7Hb] loaded module [x-pack-deprecation]
] [ZGgC7Hb] loaded module [x-pack-graph]
] [ZGgC7Hb] loaded module [x-pack-logstash]
] [ZGgC7Hb] loaded module [x-pack-ml]
] [ZGgC7Hb] loaded module [x-pack-monitoring]
] [ZGgC7Hb] loaded module [x-pack-rollup]
] [ZGgC7Hb] loaded module [x-pack-security]
] [ZGgC7Hb] loaded module [x-pack-sql]
] [ZGgC7Hb] loaded module [x-pack-upgrade]
] [ZGgC7Hb] loaded module [x-pack-watcher]
] [ZGgC7Hb] no plugins loaded
[ZGgC7Hb] parsed [0] roles from file [/home/andrea2/elasticsearch-6.3.2/config/roles.yml]
MessageHandler] [controller/2436] [Main.cc@109] controller (64 bit): Version 6.3.2 (Build 903094f295d249) Copyright (c) 2018 Elasticsearch BV
] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
] [ZGgC7Hb] using discovery type [zen]
] [ZGgC7Hb] initialized
] [ZGgC7Hb] starting ...
] [ZGgC7Hb] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
] [ZGgC7Hb] zen-disco-elected-as-master ([0] nodes joined)[, ], reason: new_master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}
[ZGgC7Hb] new_master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, reason: apply cluster state (from master [master {ZGgC7Hb}{ZGgC7HbMSw2FKAvEFH28-w}{a0kLfcSMRRGhkqvYTd6cbw}{127.0.0.1}{127.0.0.1:9300}{ml.machine_memory=4135022592, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)[, ]]])
Netty4HttpServerTransport] [ZGgC7Hb] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
] [ZGgC7Hb] started
RoleMappingStore] [ZGgC7Hb] Failed to clear cache for realms [[]]
] [ZGgC7Hb] license [e4dc900d-e45c-4547-a2c8-32a454aa57bb] mode [basic] - valid
] [ZGgC7Hb] recovered [0] indices into cluster_state
Native controller process has stopped - no new native processes can be started
] [ZGgC7Hb] stopping ...
] [ZGgC7Hb] stopping watch service, reason [shutdown initiated]
] [ZGgC7Hb] stopped
] [ZGgC7Hb] closing ...
] [ZGgC7Hb] closed

Why is it not closed?

Thanks in advance

dmitri · August 10, 2018, 5:59pm

Hi Andrea,

What operating system are you running on?

Also, is this a clean install of elasticsearch? Or have you upgraded an existing one?

Kind regards,
Dimitris

droberts195 · August 13, 2018, 3:38pm

Also, there's a large gap in the logs between 08:15:09 and 08:15:54. If you get the same problem every time you start the node, please could you try running with DEBUG level logging so that we can see if any additional messages get logged during the gap. It seems that something is timing out causing the node to shut itself down, but it's not completely clear what.

system · September 11, 2018, 10:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.