Security error after re-install of ElasticSearch

[MKirby]

Good Morning I have been receiving multiple messages that read

Error: [object Object]: shard_not_found_exception
    at http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:361224
    at async index_patterns_IndexPatternsService.refreshFieldSpecMap (http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:527636)
    at async index_patterns_IndexPatternsService.getSavedObjectAndInit (http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:530353)

This has the effect that I cannot see any data in the Discover or Dashboard.
I tried restarting the Elasticsearch service this morning without success.

Is there something that can be done in Dev Tools? Any help or direction is much appreciated.

[warkolm (Mark Walkom)]

Is there anything in your Elasticsearch logs?
What does a request to ES-HOSTNAME-IP:9200 return?

[MKirby]

Thank you for the reply Mark. When you ask what does a request to the ES-hostname-ip:9200 return with, you are refering to a curl -XGET, correct?

The response to a curl -XGET is a proper ES response ending in "tagline" : "You know, for search"

I will check the ES logs and let you know what I see.

[MKirby]

The ES log shows the following:

[WARN] [o.e.i.s.RetentionLeaseSyncAction] [es:domain] [.apm-custom-link][0] retention lease sync failed
search.transport.RemoteTransportException: [es.domain][IP:9200][indices:admin/seq_no/retention_lease_background_sync[p]]
Caused by: org.elasticsearch.gateway.WriteStateException: failed to write state to the first location tmp file /var/lib/elasticsearch/nodes/0/indices/MRLsvQkyTvqffVa2vEpTw/0/retention-leases-75.st.tmp
     at org.elasticsearch.gateway.MetadataStateFormat.writeStatetoFirstLocation(MetadataStateFormat.java:116) ~[elasticsearch-7.13.4.jar.7.13.4]
     at org.elasticsearch.gateway.MetadataStateFormat.write(MetedataStateFormat.java:232) ~[elasticsearch-7.13.4.jar:7.13.4}
   at org.elasticsearch.gateway.MetadataStateFormat.writeAndCleanup(MetadataStateFormat.java.174) ~[elasticsearch-7.13.4.jar:7.13.4]
  at org.elasticsearch.index.shard.IndexShard.persistRetentionLease(IndexShard.java:2370) ~[elasticsearch-7.13.4.jar:7.13.4]

There are other lines in the log that pertain to ActionListner and IndexShardOperationPermits.acquire.

I can grab more of the log if needed. it is a lot to type out.

Thank you for any assistance you can supply.

[MKirby]

It seems that using simply the tail command on the logs you do miss quite a lot of information. When going through the full Elasticsearch log there was a space issue going on. After moving the /var/lib/Elasticsearch and /var/log/Elasticsearch files to a larger mount point all has been corrected.

For others who may come across this, do not take the easy route, go through the entire log no matter how large it is.