Elasticsearch wont start

Timshs · July 30, 2018, 2:26pm

Hi

I'm having a single node running ES but it wont start suddently.
Logs showing:

SYSLOG: Jul 30 13:53:49 rg-app-prod-elk kibana[1495]: {"type":"log","@timestamp":"2018-07-30T13:53:49Z","tags":["warning","elasticsearch","admin"],"pid":1495,"message":"Unable to revive connection: http://localhost:9200/"}
Jul 30 13:53:49 rg-app-prod-elk kibana[1495]: {"type":"log","@timestamp":"2018-07-30T13:53:49Z","tags":["warning","elasticsearch","admin"],"pid":1495,"message":"No living connections"}

ES node log:
[2018-07-04T23:19:42,399][WARN ][o.e.c.r.a.DiskThresholdMonitor] [PROD ELK STACK] high disk watermark [90%] exceeded on [LLArx98CRD-mXivQicPXTA][PROD ELK STACK][/var/lib/elasticsearch/PROD-elasticsearch/nodes/0] free: 2.8gb[9.7%], shards will be relocated away from this node
[2018-07-04T23:19:42,399][INFO ][o.e.c.r.a.DiskThresholdMonitor] [PROD ELK STACK] rerouting shards: [high disk watermark exceeded on one or more nodes]

ES status:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Mon 2018-07-30 14:25:35 UTC; 27s ago
Docs: http://www.elastic.co
Process: 2636 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2631 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2636 (code=exited, status=1/FAILURE)
CPU: 2.569s

Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # ----------------------------------- Memory -----------------------------------
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # Lock the memory on startup:
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: bootstrap.memory_lock: true
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: #
Jul 30 14:25:35 rg-app-prod-elk elasticsearch[2636]: # Make sure that the ES_HEAP_SIZE environment variable is set to about half the memory
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Unit entered failed state.
Jul 30 14:25:35 rg-app-prod-elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

What to do ?

spinscale · August 2, 2018, 11:30am

can you paste the full error message from systemd or is this all?

--Alex

Timshs · August 3, 2018, 6:04am

This one ?

Aug 02 17:12:50 rg-app-prod-elk systemd[1]: Starting Elasticsearch...
Aug 02 17:12:50 rg-app-prod-elk systemd[1]: Started Elasticsearch.
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.FileSystemLoopException: /etc/elasticsearch/tempelasticsearch
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: Likely root cause: java.nio.file.FileSystemLoopException: /etc/elasticsearch/tempelasticsearch
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:294)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at java.nio.file.Files.walkFileTree(Files.java:2706)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:150)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.common.logging.LogConfigurator.configure(LogConfigurator.java:122)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:307)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.cli.Command.main(Command.java:90)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84)
Aug 02 17:12:54 rg-app-prod-elk elasticsearch[1815]: Refer to the log for complete error details.
Aug 02 17:12:54 rg-app-prod-elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Aug 02 17:12:54 rg-app-prod-elk systemd[1]: elasticsearch.service: Unit entered failed state.
Aug 02 17:12:54 rg-app-prod-elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

spinscale · August 3, 2018, 7:00am

Out of whatever reason you seem to have a /etc/elasticsearch/tempelasticsearch directory/link/file - you shold not have any additional files in there, except what is needed by ES. Try deleting that entity and see if that works.

Timshs · August 3, 2018, 9:54am

Still same error.

Running on 17.10 with ES 5.6.5

If you tell me exactly which logs you need to see I will find them.

spinscale · August 3, 2018, 11:18am

the systemd start up logs from the most recent try to start

Timshs · August 3, 2018, 1:35pm

[2018-08-03T12:00:52,665][WARN ][o.e.c.l.LogConfigurator [2018-08-03T12:00:52,756][WARN ][o.e.b.JNANatives [2018-08-03T12:00:52,757][WARN ][o.e.b.JNANatives [2018-08-03T12:00:52,757][WARN ][o.e.b.JNANatives [2018-08-03T12:00:52,757][WARN ][o.e.b.JNANatives # allow user 'elasticsearch' mlockall
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
[2018-08-03T12:00:52,757][WARN ][o.e.b.JNANatives [2018-08-03T12:00:52,873][INFO ][o.e.n.Node [2018-08-03T12:00:53,445][INFO ][o.e.e.NodeEnvironment [2018-08-03T12:00:53,445][INFO ][o.e.e.NodeEnvironment [2018-08-03T12:00:55,784][INFO ][o.e.n.Node [2018-08-03T12:00:55,784][INFO ][o.e.n.Node [2018-08-03T12:00:55,784][INFO ][o.e.n.Node [2018-08-03T12:00:56,509][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,509][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,509][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,509][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,509][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:56,510][INFO ][o.e.p.PluginsService [2018-08-03T12:00:57,781][INFO ][o.e.d.DiscoveryModule [2018-08-03T12:01:00,635][INFO ][o.e.n.Node [2018-08-03T12:01:00,636][INFO ][o.e.n.Node [2018-08-03T12:01:00,786][INFO ][o.e.t.TransportService [2018-08-03T12:01:00,795][INFO ][o.e.b.BootstrapChecks [2018-08-03T12:01:00,799][ERROR][o.e.b.Bootstrap [1] bootstrap checks failed
[1]: memory locking requested for elasticsearch [2018-08-03T12:01:00,802][INFO ][o.e.n.Node [2018-08-03T12:01:00,861][INFO ][o.e.n.Node [2018-08-03T12:01:00,861][INFO ][o.e.n.Node ] ignoring unsupported logging configuration file [/etc/elasticsearch/logging.yml], logging is configured via [/etc/elasticsearch/log4j2.properties]
] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
] This can result in part of the JVM being swapped out.
] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
] These can be adjusted by modifying /etc/security/limits.conf, for example:
] If you are logged in interactively, you will have to re-login for the new limits to take effect.
] [PROD ELK STACK] initializing ...
] [PROD ELK STACK] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [168.7gb], net total_space [193.6gb], spins? [possibly], types [ext4]
] [PROD ELK STACK] heap size [3.9gb], compressed ordinary object pointers [true]
] [PROD ELK STACK] node name [PROD ELK STACK], node ID [LLArx98CRD-mXivQicPXTA]
] [PROD ELK STACK] version[5.6.5], pid[1796], build[6a37571/2017-12-04T07:50:10.466Z], OS[Linux/4.10.0-42-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_151/25.151-b12]
] [PROD ELK STACK] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
] [PROD ELK STACK] loaded module [aggs-matrix-stats]
] [PROD ELK STACK] loaded module [ingest-common]
] [PROD ELK STACK] loaded module [lang-expression]
] [PROD ELK STACK] loaded module [lang-groovy]
] [PROD ELK STACK] loaded module [lang-mustache]
] [PROD ELK STACK] loaded module [lang-painless]
] [PROD ELK STACK] loaded module [parent-join]
] [PROD ELK STACK] loaded module [percolator]
] [PROD ELK STACK] loaded module [reindex]
] [PROD ELK STACK] loaded module [transport-netty3]
] [PROD ELK STACK] loaded module [transport-netty4]
] [PROD ELK STACK] no plugins loaded
] [PROD ELK STACK] using discovery type [zen]
] [PROD ELK STACK] initialized
] [PROD ELK STACK] starting ...
] [PROD ELK STACK] publish_address {10.2.128.8:9300}, bound_addresses {[::]:9300}
] [PROD ELK STACK] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
] [PROD ELK STACK] node validation exception
process but memory is not locked
] [PROD ELK STACK] stopping ...
] [PROD ELK STACK] stopped
] [PROD ELK STACK] closing ...

spinscale · August 6, 2018, 8:07am

this is the culprit

You configured memory locking, but Elasticsearch is not allowed to do this, because the operating system is not configured properly.

The last link shows what to check for.

Timshs · August 24, 2018, 1:30pm

Thank you so much for info. I got it up and running after taking the node out of the cluster I tried to setup

system · September 21, 2018, 1:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.