Elasticsearch service is crashing while indexing

Lisahtwy · May 13, 2020, 3:30pm

Hi,

I am running Elasticsearch on CentOS, and to indexing the 5GB files automatically using fsCrawler commands in a python script.When I am running that script, elastic search is crashing. below is the status using systemctl

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Wed 2020-05-13 14:52:54 UTC; 3min 3s ago
     Docs: http://www.elastic.co
  Process: 2318 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
 Main PID: 2318 (code=killed, signal=KILL)

May 13 03:45:02 li393-89.members.linode.com systemd[1]: Starting Elasticsearch...
May 13 03:45:05 li393-89.members.linode.com elasticsearch[2318]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be re...e release.
May 13 03:45:36 li393-89.members.linode.com systemd[1]: Started Elasticsearch.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
May 13 14:52:54 li393-89.members.linode.com systemd[1]: Unit elasticsearch.service entered failed state.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Here is one of the warning logs i am getting in /var/log/elasticsearch/elasticsearch.log

[2020-05-13T14:56:35,502][INFO ][o.e.c.s.ClusterApplierService] [li393-89.members.linode.com] master node changed {previous [], current [{li393-89.members.linode.com}{LOrZIItxSaChVbGzf11YCQ}{QD_LgkwuTIGp5ChxR5L2nA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8331206656, xpack.installed=true, ml.max_open_jobs=20}]}, term: 11, version: 1833, reason: Publication{term=11, version=1833}
[2020-05-13T14:56:35,669][INFO ][o.e.h.AbstractHttpServerTransport] [li393-89.members.linode.com] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-05-13T14:56:35,670][INFO ][o.e.n.Node               ] [li393-89.members.linode.com] started
[2020-05-13T14:56:36,523][WARN ][r.suppressed             ] [li393-89.members.linode.com] path: /.kibana/_search, params: {index=.kibana, filter_path=hits.hits._id,hits.hits._source}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:175) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:467) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:400) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:212) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:242) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:87) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.action.search.HttpChannelTaskHandler.execute(HttpChannelTaskHandler.java:60) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:114) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.1.jar:7.5.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.1.jar:7.5.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]

Lisahtwy · May 13, 2020, 3:30pm

script:

try:
                proc1 = Popen(['bin/fscrawler', '--config_dir', dir_index, index, '--loop', '1' ],stdout=PIPE, stdin=PIPE, stderr=PIPE,universal_newlines=True)
               
                out1,err1 = proc1.communicate(input="{}\n".format("Y"))

                print("out1", out1)
                #print(err1)
            except Exception as e:
                print("Exception: ", e)    

            settings_path = str(working_dir)+str(dir_index)+"/"+str(index)

            os.chdir(settings_path)
            p1 = subprocess.run(["ls", "-l",settings_path], stdout=PIPE, stderr=PIPE)
            #print(p1.stdout.decode())

            with open('_settings.yaml')as f:
                list_settings= yaml.safe_load(f)
            #print(list_settings["fs"]["url"])

            list_settings["fs"]["url"] = row['folder_path']
            list_settings["elasticsearch"]["bulk_size"]= 5
            list_settings['server']= {}
            list_settings["server"]["hostname"] = row['ip']
            list_settings["server"]["username"] = row['username']
            list_settings["server"]["password"] = row['password']
            list_settings["server"]["port"] = row['port']
            list_settings["server"]["protocol"] = protocol

            with open("_settings.yaml","r+") as f:
                yaml.dump(list_settings, f)

             os.chdir(working_dir)

            try:
                proc2 = Popen(['bin/fscrawler', '--config_dir', dir_index, index, '--loop', '1', '--restart' ],stdout=PIPE, stdin=PIPE, stderr=PIPE,universal_newlines=True)
         
                out2,err2 = proc2.communicate(input="{}\n".format("Y"))
                print("out2", out2)
            except Exception as e:
                #print("Exception: ", e)

Could you please tell me why Elasticsearch is crashing while running the script? I did not see any ES failing while indexing manually or using this script in development mode. The same scripts and same server with ES,indexes and whole application(flask) in development mode is working fine. Only in production I am facing this issue.
when ES is down, I restarted it using systemctl restart elasticsearch 2-3 times. after that ES didnot come up, had to reboot the server.

-Lisa

dadoonet · May 13, 2020, 3:40pm

Interesting.

Is elasticsearch really stopping?

Could you share the full elasticsearch logs?
Also the full fscrawler logs?

Lisahtwy · May 13, 2020, 4:40pm

Paste bin - ES logs

https://pastebin.com/1u72Zihk

where to find fscrawler logs?

dadoonet · May 14, 2020, 7:37am

Logs are sent to stdout.

dadoonet · May 14, 2020, 10:23am

I read the logs. Some comments.
You are using 4gb of HEAP. May be are you sending too big bulks from FSCrawler and that creates some memory pressure?
You are using 7.5. Is it possible to switch to 7.7?

Could you share the logs from /var/log/elasticsearch/hs_err_pidXXXX.log?

system · June 11, 2020, 10:23am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.