Elastic crash while import

Elastic Stack Elasticsearch
1

I have Elasticsearch installed so I am playing with it. I made a bash script which generates simple documents with name, title, perex and text fields (text is 100 sentences of lorem ipsum) and send it via _bulk api to Elasticsearch. I run 5 parallel scripts to do it and one bulk contains 250 documents. I had about 100 000 items indexed when Elasticsearch crashed and I need to manually restart the process. The index definition is here:

PUT /test2
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": { 
    "properties": {
      "name" : {
        "type" : "text",
        "fields" : {
          "raw" : {
            "type" : "keyword"
            }
        }
      },
      "title" : {
        "type" : "text",
        "fields" : {
          "raw" : {
            "type" : "keyword"
          }
        }
      },
      "perex" : {
        "type" : "text"
      },
      "text" : {
        "type" : "text"
      }
    }    
  }
}

elasticsearch.log file contains this:

[2021-07-01T20:58:45,424][INFO ][o.e.t.NettyAllocator     ] [vmi503579.contaboserver.net] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=4mb}]
[2021-07-01T20:58:45,561][INFO ][o.e.d.DiscoveryModule    ] [vmi503579.contaboserver.net] using discovery type [zen] and seed hosts providers [settings]
[2021-07-01T20:58:46,533][INFO ][o.e.g.DanglingIndicesState] [vmi503579.contaboserver.net] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2021-07-01T20:58:47,422][INFO ][o.e.n.Node               ] [vmi503579.contaboserver.net] initialized
[2021-07-01T20:58:47,424][INFO ][o.e.n.Node               ] [vmi503579.contaboserver.net] starting ...
[2021-07-01T20:58:47,447][INFO ][o.e.x.s.c.f.PersistentCache] [vmi503579.contaboserver.net] persistent cache index loaded
[2021-07-01T20:58:47,699][INFO ][o.e.t.TransportService   ] [vmi503579.contaboserver.net] publish_address {localhost/127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2021-07-01T20:58:48,587][WARN ][o.e.b.BootstrapChecks    ] [vmi503579.contaboserver.net] the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2021-07-01T20:58:48,589][INFO ][o.e.c.c.Coordinator      ] [vmi503579.contaboserver.net] cluster UUID [QR6cWTx6ToSKgVEAqs34TQ]
[2021-07-01T20:58:48,605][INFO ][o.e.c.c.ClusterBootstrapService] [vmi503579.contaboserver.net] no discovery configuration found, will perform best-effort cluster bootstrapping after [3s] unless existing master is discovered
[2021-07-01T20:58:48,719][INFO ][o.e.c.s.MasterService    ] [vmi503579.contaboserver.net] elected-as-master ([1] nodes joined)[{vmi503579.contaboserver.net}{Wp9M32ZRR_mNKO29Ttmaew}{ssVTNtcLSraJu-hzJiUzNA}{localhost}{127.0.0.1:9300}{cdfhilmrstw} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 8, version: 215, delta: master node changed {previous [], current [{vmi503579.contaboserver.net}{Wp9M32ZRR_mNKO29Ttmaew}{ssVTNtcLSraJu-hzJiUzNA}{localhost}{127.0.0.1:9300}{cdfhilmrstw}]}
[2021-07-01T20:58:48,880][INFO ][o.e.c.s.ClusterApplierService] [vmi503579.contaboserver.net] master node changed {previous [], current [{vmi503579.contaboserver.net}{Wp9M32ZRR_mNKO29Ttmaew}{ssVTNtcLSraJu-hzJiUzNA}{localhost}{127.0.0.1:9300}{cdfhilmrstw}]}, term: 8, version: 215, reason: Publication{term=8, version=215}
[2021-07-01T20:58:48,956][INFO ][o.e.h.AbstractHttpServerTransport] [vmi503579.contaboserver.net] publish_address {localhost/127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2021-07-01T20:58:48,957][INFO ][o.e.n.Node               ] [vmi503579.contaboserver.net] started
[2021-07-01T20:58:49,874][INFO ][o.e.l.LicenseService     ] [vmi503579.contaboserver.net] license [f35cccd9-e198-4bb4-baeb-1c7acf80cea4] mode [basic] - valid
[2021-07-01T20:58:49,876][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [vmi503579.contaboserver.net] Active license is now [BASIC]; Security is disabled
[2021-07-01T20:58:49,877][WARN ][o.e.x.s.s.SecurityStatusChangeListener] [vmi503579.contaboserver.net] Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.13/security-minimal-setup.html to enable security.
[2021-07-01T20:58:49,883][INFO ][o.e.g.GatewayService     ] [vmi503579.contaboserver.net] recovered [8] indices into cluster_state
[2021-07-01T20:58:51,368][WARN ][r.suppressed             ] [vmi503579.contaboserver.net] path: /.kibana_task_manager/_search, params: {ignore_unavailable=true, index=.kibana_task_manager, track_total_hits=true}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:661) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:384) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:694) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:467) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.access$000(AbstractSearchAsyncAction.java:62) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:316) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.ActionListener$Delegating.onFailure(ActionListener.java:66) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:400) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TransportService$5.handleException(TransportService.java:738) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1283) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1392) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1366) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:50) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.transport.TransportChannel.sendErrorResponse(TransportChannel.java:45) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:40) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:77) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.onFailure(ThreadContext.java:720) [elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:28) [elasticsearch-7.13.2.jar:7.13.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.action.NoShardAvailableActionException: [vmi503579.contaboserver.net][127.0.0.1:9300][indices:data/read/search[phase/query]]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:500) ~[elasticsearch-7.13.2.jar:7.13.2]
	at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:449) [elasticsearch-7.13.2.jar:7.13.2]
	... 18 more

Here are the links to log files:
https://drive.google.com/file/d/1kQQyF6d7HX_UmYpgVplTTcDREwHLpRPN/view?usp=sharing
https://drive.google.com/file/d/1QgbRSJA67_2XM4WfJgLBfBksSfwaGTje/view?usp=sharing
https://drive.google.com/file/d/1iyzSCJ-_UqmCbHS60JvL4VaZ7mcz79y1/view?usp=sharing

Does anybody know what is the problem? I would like to fix it, but I am only a beginner in this.

2

I am not seeing any crash in the log files, just Elasticsearch running or restarting, trying to recover and then something restarting Elasticsearch.

3

I have a run small cron script every minute to check if Elasticsearch in running.

#!/bin/bash

# -c returns number of lines in grep result
isActive=$(systemctl status elasticsearch | grep "active (running)" -c)

if (( $isActive == 0 )) 
then

  systemctl start elasticsearch
  
  timestamp=$(date +"%Y-%m-%d %H-%M-%S") 
  touch /root/custom-scripts/elasticsearch/start.log
  echo "${timestamp} Elasticsearch service has to be restarted by /root/custom-scripts/elasticsearch/start-elasticsearch.sh because service status was not active." >> /root/custom-scripts/elasticsearch/start.log

fi

Is it possible the condition active (running) is not correct? I see this script starts Elastic on average three times per hour.

Here are today logs.
Elasticsearch.log
and gc.log

May be there are not enough resources. It is VPS with 6vCPU 8GB ram 400GB SSD. There is only one node and few indices. One index with 950 000 items.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.