Hi,
When I am indexing few repositories files using fscrawler, elasticsearch is crashing.
The exception i got from my program is :
ConnectionError(<urllib3.connection.HTTPConnection object at 0x7feaf2a65e10>: Failed to establish a new connection:
[Errno 111] Connection refused) caused by:
NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7feaf2a65e10>:
Failed to establish a new connection: [Errno 111] Connection refused)
The status of elastic search is:
elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since Wed 2020-05-27 21:48:11 UTC; 59s ago
Docs: http://www.elastic.co
Process: 3499 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 3499 (code=killed, signal=KILL)
CGroup: /system.slice/elasticsearch.service
May 27 21:25:26 li393-89.members.linode.com systemd[1]: Starting Elasticsearch...
May 27 21:25:27 li393-89.members.linode.com elasticsearch[3499]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be re...e release.
May 27 21:25:51 li393-89.members.linode.com systemd[1]: Started Elasticsearch.
May 27 21:48:11 li393-89.members.linode.com systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
May 27 21:48:11 li393-89.members.linode.com systemd[1]: Unit elasticsearch.service entered failed state.
May 27 21:48:11 li393-89.members.linode.com systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
In /var/log/elasticsearch/elasticsearch.log , i see below WARNING logs
[2020-05-27T22:30:31,247][WARN ][r.suppressed ] [li393-89.members.linode.com] path: /.kibana_task_manager/_update_by_query, params: {ignore_unavailable=true, refresh=true, conflicts=proceed, index=.kibana_task_manager, max_docs=10}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:534) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:305) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-7.5.1.jar:7.5.1]
[2020-05-27T22:30:32,085][WARN ][r.suppressed ] [li393-89.members.linode.com] path: /.kibana/_doc/space%3Adefault, params: {index=.kibana, id=space:default}
org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][_doc][space:default]: routing [null]]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:224) [elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:201) [elasticsearch-7.5.1.jar:7.5.1]
Below is the WARNING log I saw when elasticsearch crashed for the first time.
[2020-05-27T14:22:38,029][WARN ][r.suppressed ] [li393-89.members.linode.com] path: /.kibana_task_manager/_update_by_query, params: {ignore_unavailable=true, refresh=true, conflicts=proceed, index=.kibana_task_manager, max_docs=10}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:189) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:175) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:467) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:400) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:212) ~[elasticsearch-7.5.1.jar:7.5.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.1.jar:7.5.1]
Below is my elasticsearch.yml file
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
discovery.seed_hosts: ["50.116.48.89:9200"]
If I delete all my indexes and restart the elasticsearch , then crash is not happening. If I run without these two steps, elasticsearch is failing in middle of my indexing.
Could you please tell me why the crash is happening ??
-Lisa
