Hi,
I am running Elasticsearch on CentOS, and to indexing the 5GB files automatically using fsCrawler commands in a python script.When I am running that script, elastic search is crashing. below is the status using systemctl
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2020-05-13 14:52:54 UTC; 3min 3s ago
Docs: http://www.elastic.co
Process: 2318 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 2318 (code=killed, signal=KILL)
May 13 03:45:02 li393-89.members.linode.com systemd[1]: Starting Elasticsearch...
May 13 03:45:05 li393-89.members.linode.com elasticsearch[2318]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be re...e release.
May 13 03:45:36 li393-89.members.linode.com systemd[1]: Started Elasticsearch.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
May 13 14:52:54 li393-89.members.linode.com systemd[1]: Unit elasticsearch.service entered failed state.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
Here is one of the warning logs i am getting in /var/log/elasticsearch/elasticsearch.log
[2020-05-13T14:56:35,502][INFO ][o.e.c.s.ClusterApplierService] [li393-89.members.linode.com] master node changed {previous [], current [{li393-89.members.linode.com}{LOrZIItxSaChVbGzf11YCQ}{QD_LgkwuTIGp5ChxR5L2nA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8331206656, xpack.installed=true, ml.max_open_jobs=20}]}, term: 11, version: 1833, reason: Publication{term=11, version=1833}
[2020-05-13T14:56:35,669][INFO ][o.e.h.AbstractHttpServerTransport] [li393-89.members.linode.com] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-05-13T14:56:35,670][INFO ][o.e.n.Node ] [li393-89.members.linode.com] started
[2020-05-13T14:56:36,523][WARN ][r.suppressed ] [li393-89.members.linode.com] path: /.kibana/_search, params: {index=.kibana, filter_path=hits.hits._id,hits.hits._source}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:175) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:467) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:400) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:212) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:242) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:87) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.action.search.HttpChannelTaskHandler.execute(HttpChannelTaskHandler.java:60) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:114) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.1.jar:7.5.1]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.1.jar:7.5.1]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.1.jar:7.5.1]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:830) [?:?]