Unable start elasticsearch

karthick_kn · December 4, 2021, 2:54am

here I attached my log

 sudo tail -f /var/log/elasticsearch/my-application.log
[2021-12-03T17:10:26,220][INFO ][o.e.x.w.WatcherService   ] [node-1] stopping watch service, reason [shutdown initiated]
[2021-12-03T17:10:26,220][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/3483460] [Main.cc@169] ML controller exiting
[2021-12-03T17:10:26,221][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
[2021-12-03T17:10:26,222][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2021-12-03T17:10:26,444][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
[2021-12-03T17:10:26,994][INFO ][o.e.c.m.MetadataIndexTemplateService] [node-1] adding template [.ml-state] for index patterns [.ml-state*]
[2021-12-03T17:10:27,220][INFO ][o.e.n.Node               ] [node-1] stopped
[2021-12-03T17:10:27,220][INFO ][o.e.n.Node               ] [node-1] closing ...
[2021-12-03T17:10:27,234][WARN ][o.e.s.SystemdPlugin      ] [node-1] sd_notify returned error [-13]
[2021-12-03T17:10:27,244][INFO ][o.e.n.Node               ] [node-1] closed

[Kalinga@kalinga ~]$ sudo systemctl status Elasticsearch

Elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/Elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sat 2021-12-04 08:30:39 IST; 15s ago
Docs: https://www.elastic.co
Process: 3578506 ExecStart=/usr/share/Elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/Elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 3578506 (code=exited, status=1/FAILURE)

Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.cli.Command.main(Command.java:79)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: at org.Elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Dec 04 08:30:38 kalinga.coreservices01 systemd-entrypoint[3578506]: For complete error details, refer to the log at /var/log/Elasticsearch/my-application.log
Dec 04 08:30:39 kalinga.coreservices01 systemd[1]: Elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Dec 04 08:30:39 kalinga.coreservices01 systemd[1]: Elasticsearch.service: Failed with result 'exit-code'.
Dec 04 08:30:39 kalinga.coreservices01 systemd[1]: Failed to startElasticsearch.

t[Kalinga@kalinga ~]$ grep -E -i 'erro|warn|fail|expir' /var/log/elasticsearch/my-application.log
[2021-12-04T08:30:12,521][INFO ][o.e.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -Djava.io.tmpdir=/var/log/elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=2147483648, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2021-12-04T08:30:38,295][WARN ][o.e.d.SeedHostsResolver  ] [node-1] failed to resolve host [host1]
[2021-12-04T08:30:38,305][WARN ][o.e.d.SeedHostsResolver  ] [node-1] failed to resolve host [host2]
[2021-12-04T08:30:38,523][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.RuntimeException: sd_notify returned error [-13]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
[2021-12-04T08:30:38,525][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: sd_notify returned error [-13]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
Caused by: java.lang.RuntimeException: sd_notify returned error [-13]
[2021-12-04T08:30:39,224][WARN ][o.e.s.SystemdPlugin      ] [node-1] sd_notify returned error [-13]
ype or paste code here
``

warkolm · December 5, 2021, 11:48pm

Can you please show more of your Elasticsearch log.

karthick_kn · December 6, 2021, 4:44am

type [2021-12-06T09:50:41,196][INFO ][o.e.e.NodeEnvironment    ] [node-1] heap size [16gb], compressed ordinary object pointers [true]
[2021-12-06T09:50:41,544][INFO ][o.e.n.Node               ] [node-1] node name [node-1], node ID [M6uJAqIHS7y45Spa1qzueg], cluster name [my-application], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2021-12-06T09:50:49,949][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/3857213] [Main.cc@117] controller (64 bit): Version 7.11.2 (Build c940034b34c11c) Copyright (c) 2021 Elasticsearch BV
[2021-12-06T09:50:50,843][INFO ][o.e.x.s.a.s.FileRolesStore] [node-1] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2021-12-06T09:50:53,347][INFO ][o.e.t.NettyAllocator     ] [node-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=8mb}]
[2021-12-06T09:50:53,482][INFO ][o.e.d.DiscoveryModule    ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2021-12-06T09:50:54,386][INFO ][o.e.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-12-06T09:50:55,209][INFO ][o.e.n.Node               ] [node-1] initialized
[2021-12-06T09:50:55,210][INFO ][o.e.n.Node               ] [node-1] starting ...
[2021-12-06T09:50:55,232][INFO ][o.e.x.s.c.PersistentCache] [node-1] persistent cache index loaded
[2021-12-06T09:50:55,455][INFO ][o.e.t.TransportService   ] [node-1] publish_address {209.182.237.225:9300}, bound_addresses {[::]:9300}
[2021-12-06T09:50:55,755][INFO ][o.e.b.BootstrapChecks    ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-12-06T09:50:55,760][INFO ][o.e.c.c.Coordinator      ] [node-1] cluster UUID [KYxAioBUTYW2eyYN-nsAvw]
[2021-12-06T09:50:55,787][WARN ][o.e.d.SeedHostsResolver  ] [node-1] failed to resolve host [host1]
java.net.UnknownHostException: host1: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
        at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:932) ~[?:?]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1517) ~[?:?]
        at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:851) ~[?:?]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:545) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:487) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:967) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:133) ~[elasticsearch-7.11.2.jar:7.11.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.11.2.jar:7.11.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-12-06T09:50:55,797][WARN ][o.e.d.SeedHostsResolver  ] [node-1] failed to resolve host [host2]
java.net.UnknownHostException: host2: Name or service not known
        at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
        at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:932) ~[?:?]
        at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1517) ~[?:?]
        at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:851) ~[?:?]
        at java.net.InetAddress.getAllByName0(InetAddress.java:1507) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1366) ~[?:?]
        at java.net.InetAddress.getAllByName(InetAddress.java:1300) ~[?:?]
        at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:545) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:487) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:967) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:133) ~[elasticsearch-7.11.2.jar:7.11.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.11.2.jar:7.11.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-12-06T09:50:55,909][INFO ][o.e.c.s.MasterService    ] [node-1] elected-as-master ([1] nodes joined)[{node-1}{M6uJAqIHS7y45Spa1qzueg}{D01N4yriQc6-EjM9t1JChA}{209.182.237.225}{209.182.237.225:9300}{cdhilmrstw}{ml.machine_memory=50423828480, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=17179869184} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 10, version: 26, delta: master node changed {previous [], current [{node-1}{M6uJAqIHS7y45Spa1qzueg}{D01N4yriQc6-EjM9t1JChA}{209.182.237.225}{209.182.237.225:9300}{cdhilmrstw}{ml.machine_memory=50423828480, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=17179869184}]}
[2021-12-06T09:50:55,978][INFO ][o.e.c.s.ClusterApplierService] [node-1] master node changed {previous [], current [{node-1}{M6uJAqIHS7y45Spa1qzueg}{D01N4yriQc6-EjM9t1JChA}{209.182.237.225}{209.182.237.225:9300}{cdhilmrstw}{ml.machine_memory=50423828480, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=17179869184}]}, term: 10, version: 26, reason: Publication{term=10, version=26}
[2021-12-06T09:50:56,055][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {209.182.237.225:9200}, bound_addresses {[::]:9200}
[2021-12-06T09:50:56,056][INFO ][o.e.n.Node               ] [node-1] started
[2021-12-06T09:50:56,061][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.RuntimeException: sd_notify returned error [-13]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:129) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:909) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) [elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) [elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) [elasticsearch-7.11.2.jar:7.11.2]
[2021-12-06T09:50:56,062][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: sd_notify returned error [-13]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:116) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.cli.Command.main(Command.java:79) ~[elasticsearch-cli-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81) ~[elasticsearch-7.11.2.jar:7.11.2]
Caused by: java.lang.RuntimeException: sd_notify returned error [-13]
        at org.elasticsearch.systemd.SystemdPlugin.onNodeStarted(SystemdPlugin.java:129) ~[?:?]
        at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
        at org.elasticsearch.node.Node.start(Node.java:909) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:310) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:396) ~[elasticsearch-7.11.2.jar:7.11.2]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-7.11.2.jar:7.11.2]
        ... 6 more
[2021-12-06T09:50:56,071][INFO ][o.e.n.Node               ] [node-1] stopping ...
[2021-12-06T09:50:56,076][INFO ][o.e.x.w.WatcherService   ] [node-1] stopping watch service, reason [shutdown initiated]
[2021-12-06T09:50:56,077][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2021-12-06T09:50:56,077][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/3857213] [Main.cc@169] ML controller exiting
[2021-12-06T09:50:56,078][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started
[2021-12-06T09:50:56,215][INFO ][o.e.g.GatewayService     ] [node-1] recovered [0] indices into cluster_state
[2021-12-06T09:50:56,660][INFO ][o.e.c.m.MetadataIndexTemplateService] [node-1] adding component template [metrics-mappings]
[2021-12-06T09:50:56,828][INFO ][o.e.n.Node               ] [node-1] stopped
[2021-12-06T09:50:56,829][INFO ][o.e.n.Node               ] [node-1] closing ...
[2021-12-06T09:50:56,838][WARN ][o.e.s.SystemdPlugin      ] [node-1] sd_notify returned error [-13]
[2021-12-06T09:50:56,842][INFO ][o.e.n.Node               ] [node-1] closed
or paste code here

warkolm · December 6, 2021, 4:47am

What does your config look like?

can.ozdemir · December 6, 2021, 5:57am

It looks like your node cannot start up sd_notify plugin properly, do you know what this plugin is, have you upgraded your cluster lately and forgot to upgrade the plugin as well?

Can you try removing the plugin from your node and start it like that?

Best regards

system · January 3, 2022, 5:57am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.