Good Morning I have been receiving multiple messages that read
Error: [object Object]: shard_not_found_exception
at http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:361224
at async index_patterns_IndexPatternsService.refreshFieldSpecMap (http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:527636)
at async index_patterns_IndexPatternsService.getSavedObjectAndInit (http://10.12.36.50:5601/41022/bundles/plugin/data/kibana/data.plugin.js:1:530353)
This has the effect that I cannot see any data in the Discover or Dashboard.
I tried restarting the Elasticsearch service this morning without success.
Is there something that can be done in Dev Tools? Any help or direction is much appreciated.
[WARN] [o.e.i.s.RetentionLeaseSyncAction] [es:domain] [.apm-custom-link][0] retention lease sync failed
search.transport.RemoteTransportException: [es.domain][IP:9200][indices:admin/seq_no/retention_lease_background_sync[p]]
Caused by: org.elasticsearch.gateway.WriteStateException: failed to write state to the first location tmp file /var/lib/elasticsearch/nodes/0/indices/MRLsvQkyTvqffVa2vEpTw/0/retention-leases-75.st.tmp
at org.elasticsearch.gateway.MetadataStateFormat.writeStatetoFirstLocation(MetadataStateFormat.java:116) ~[elasticsearch-7.13.4.jar.7.13.4]
at org.elasticsearch.gateway.MetadataStateFormat.write(MetedataStateFormat.java:232) ~[elasticsearch-7.13.4.jar:7.13.4}
at org.elasticsearch.gateway.MetadataStateFormat.writeAndCleanup(MetadataStateFormat.java.174) ~[elasticsearch-7.13.4.jar:7.13.4]
at org.elasticsearch.index.shard.IndexShard.persistRetentionLease(IndexShard.java:2370) ~[elasticsearch-7.13.4.jar:7.13.4]
There are other lines in the log that pertain to ActionListner and IndexShardOperationPermits.acquire.
I can grab more of the log if needed. it is a lot to type out.
It seems that using simply the tail command on the logs you do miss quite a lot of information. When going through the full Elasticsearch log there was a space issue going on. After moving the /var/lib/Elasticsearch and /var/log/Elasticsearch files to a larger mount point all has been corrected.
For others who may come across this, do not take the easy route, go through the entire log no matter how large it is.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.