# Elasticsearch service is crashing while indexing

**URL:** <https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472>\
**Category:** Elasticsearch\
**Created:** [May 13, 2020, 3:30pm UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472 "2020-05-13T15:30:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lisahtwy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisahtwy/32/53910_2.png) [@Lisahtwy](https://discuss.elastic.co/u/Lisahtwy)\
**Post date:** [May 13, 2020, 3:30pm UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/1 "2020-05-13T15:30:26Z")

</div>

Hi,

I am running Elasticsearch on CentOS, and to indexing the 5GB files automatically using fsCrawler commands in a python script.When I am running that script, elastic search is crashing. below is the status using systemctl

```auto
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Wed 2020-05-13 14:52:54 UTC; 3min 3s ago
     Docs: http://www.elastic.co
  Process: 2318 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
 Main PID: 2318 (code=killed, signal=KILL)

May 13 03:45:02 li393-89.members.linode.com systemd[1]: Starting Elasticsearch...
May 13 03:45:05 li393-89.members.linode.com elasticsearch[2318]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be re...e release.
May 13 03:45:36 li393-89.members.linode.com systemd[1]: Started Elasticsearch.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
May 13 14:52:54 li393-89.members.linode.com systemd[1]: Unit elasticsearch.service entered failed state.
May 13 14:52:54 li393-89.members.linode.com systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

```

Here is one of the warning logs i am getting in /var/log/elasticsearch/elasticsearch.log

```auto
[2020-05-13T14:56:35,502][INFO][o.e.c.s.ClusterApplierService] [li393-89.members.linode.com] master node changed {previous [], current [{li393-89.members.linode.com}{LOrZIItxSaChVbGzf11YCQ}{QD_LgkwuTIGp5ChxR5L2nA}{127.0.0.1}{127.0.0.1:9300}{dilm}{ml.machine_memory=8331206656, xpack.installed=true, ml.max_open_jobs=20}]}, term: 11, version: 1833, reason: Publication{term=11, version=1833}
[2020-05-13T14:56:35,669][INFO][o.e.h.AbstractHttpServerTransport] [li393-89.members.linode.com] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}
[2020-05-13T14:56:35,670][INFO][o.e.n.Node] [li393-89.members.linode.com] started
[2020-05-13T14:56:36,523][WARN][r.suppressed] [li393-89.members.linode.com] path: /.kibana/_search, params: {index=.kibana, filter_path=hits.hits._id,hits.hits._source}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:175) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:467) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:400) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:212) ~[elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:242) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:87) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:123) [x-pack-security-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:151) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.action.search.HttpChannelTaskHandler.execute(HttpChannelTaskHandler.java:60) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.action.search.RestSearchAction.lambda$prepareRequest$2(RestSearchAction.java:114) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:69) [x-pack-security-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.1.jar:7.5.1]
        at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.1.jar:7.5.1]
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.1.jar:7.5.1]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:326) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:300) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
        at java.lang.Thread.run(Thread.java:830) [?:?]

```

---

<div class="post-metadata">

**Author:** ![Lisahtwy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisahtwy/32/53910_2.png) [@Lisahtwy](https://discuss.elastic.co/u/Lisahtwy)\
**Post date:** [May 13, 2020, 3:30pm UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/2 "2020-05-13T15:30:57Z")

</div>

script:

```auto
try:
                proc1 = Popen(['bin/fscrawler', '--config_dir', dir_index, index, '--loop', '1'],stdout=PIPE, stdin=PIPE, stderr=PIPE,universal_newlines=True)
               
                out1,err1 = proc1.communicate(input="{}\n".format("Y"))

                print("out1", out1)
                #print(err1)
            except Exception as e:
                print("Exception: ", e)    

            settings_path = str(working_dir)+str(dir_index)+"/"+str(index)

            os.chdir(settings_path)
            p1 = subprocess.run(["ls", "-l",settings_path], stdout=PIPE, stderr=PIPE)
            #print(p1.stdout.decode())

            with open('_settings.yaml')as f:
                list_settings= yaml.safe_load(f)
            #print(list_settings["fs"]["url"])

            list_settings["fs"]["url"] = row['folder_path']
            list_settings["elasticsearch"]["bulk_size"]= 5
            list_settings['server']= {}
            list_settings["server"]["hostname"] = row['ip']
            list_settings["server"]["username"] = row['username']
            list_settings["server"]["password"] = row['password']
            list_settings["server"]["port"] = row['port']
            list_settings["server"]["protocol"] = protocol

            with open("_settings.yaml","r+") as f:
                yaml.dump(list_settings, f)

             os.chdir(working_dir)

            try:
                proc2 = Popen(['bin/fscrawler', '--config_dir', dir_index, index, '--loop', '1', '--restart'],stdout=PIPE, stdin=PIPE, stderr=PIPE,universal_newlines=True)
         
                out2,err2 = proc2.communicate(input="{}\n".format("Y"))
                print("out2", out2)
            except Exception as e:
                #print("Exception: ", e)

```

Could you please tell me why Elasticsearch is crashing while running the script? I did not see any ES failing while indexing manually or using this script in development mode. The same scripts and same server with ES,indexes and whole application(flask) in development mode is working fine. Only in production I am facing this issue.  
when ES is down, I restarted it using `systemctl restart elasticsearch` 2-3 times. after that ES didnot come up, had to reboot the server.

-Lisa

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 13, 2020, 3:40pm UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/3 "2020-05-13T15:40:16Z")

</div>

Interesting.

Is elasticsearch really stopping?

Could you share the full elasticsearch logs?  
Also the full fscrawler logs?

---

<div class="post-metadata">

**Author:** ![Lisahtwy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lisahtwy/32/53910_2.png) [@Lisahtwy](https://discuss.elastic.co/u/Lisahtwy)\
**Post date:** [May 13, 2020, 4:40pm UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/4 "2020-05-13T16:40:16Z")

</div>

Paste bin - ES logs

[https://pastebin.com/1u72Zihk](https://pastebin.com/1u72Zihk)

where to find fscrawler logs?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 14, 2020, 7:37am UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/5 "2020-05-14T07:37:33Z")

</div>

Logs are sent to stdout.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 14, 2020, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/6 "2020-05-14T10:23:15Z")

</div>

I read the logs. Some comments.  
You are using 4gb of HEAP. May be are you sending too big bulks from FSCrawler and that creates some memory pressure?  
You are using 7.5. Is it possible to switch to 7.7?

Could you share the logs from `/var/log/elasticsearch/hs_err_pidXXXX.log`?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 11, 2020, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-service-is-crashing-while-indexing/232472/7 "2020-06-11T10:23:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.