# Logs do not displayed in ELK. What is the problem? How to fix the error?

**URL:** https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058
**Category:** Elasticsearch
**Created:** [September 11, 2017, 1:05pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058 "2017-09-11T13:05:09Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 11, 2017, 1:05pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/1 "2017-09-11T13:05:09Z")

</div>

Hello.

Kibana version: 5.4.2  
Elasticsearch version: 5.4.2  
Logstash version: 5.4.2  
Server OS version: Linux Red Hat, docker container kibana, logstash, elasticsearch and curator.  
Logs do not displayed in ELK. What is the problem? How to fix the error?

* * *

Discover: Request to Elasticsearch failed: {"error":{"root\_cause":[],"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":[]},"status":503}  
Less

• Error: Request to Elasticsearch failed: {"error":{"root\_cause":[],"type":"search\_phase\_execution\_exception","reason":"all shards failed","phase":"query","grouped":true,"failed\_shards":[]},"status":503}  
• at [http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10760](http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10760)  
• at Function.Promise.try ([http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:22203](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:22203))  
• at [http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21573](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21573)  
• at Array.map (native)  
• at Function.Promise.map ([http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21528](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21528))  
• at callResponseHandlers ([http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10376](http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10376))  
• at [http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:27:29944](http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:27:29944)  
• at processQueue ([http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23621](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23621))  
• at [http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23888](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23888)  
• at Scope.$eval ([http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:39:4619](http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:39:4619))

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2017, 1:09pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/2 "2017-09-11T13:09:24Z")

</div>

It seems to be a problem with Elasticsearch. Is there anything in the Elasticsearch logs?

---

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 11, 2017, 1:22pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/3 "2017-09-11T13:22:14Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> It seems to be a problem with Elasticsearch. Is there anything in the Elasticsearch logs?

We perform load testing of the java application. From java application should be logs in the stream stdout. The docker log-driver gelf must send them to logstash.  
But there are no logs in elasticsearch journals.

* * *

[http://10.118.41.32:9200/\_cat/indices?v](http://10.118.41.32:9200/_cat/indices?v)

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
yellow open .kibana NrnfhK-\_TIKoow0beow72w 1 1 2 0 10.6kb 10.6kb  
red open messenger-2017.09.10 3AHvYolPQraNW-eGazlZZQ 5 1

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2017, 1:27pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/4 "2017-09-11T13:27:53Z")

</div>

Have you looked for logs in the log directory Elasticsearch has configured?

It looks like you have a problem with the `messenger-2017.09.10` index as it is in `red` state. The logs should hold more details.

---

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 11, 2017, 1:32pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/5 "2017-09-11T13:32:47Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> messenger-2017.09.10

I'm looking for logs through kibana. There are no logs in the kibana

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2017, 1:34pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/6 "2017-09-11T13:34:03Z")

</div>

Elasticsearch logs are not available through Kibana, so you need to look in the file system.

---

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 11, 2017, 1:36pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/7 "2017-09-11T13:36:47Z")

</div>

> [@Christian\_Dahlqvist](#):
>
> Elasticsearch logs are not available through Kibana, so you need to look in the file system.

Tell me please, in what elasticsearch file are the logs stored?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2017, 1:48pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/8 "2017-09-11T13:48:30Z")

</div>

As you are using docker, they might be accessible via [docker logs](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#_inspect_status_of_cluster).

---

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 11, 2017, 3:49pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/9 "2017-09-11T15:49:21Z")

</div>

{"log":"[2017-09-10T09:23:45,416][WARN][o.e.i.e.Engine] [ywpbCov] [messenger-2017.09.10][1] failed engine [merge failed]\n","stream":"stdout","time":"2017-09-10T09:23:45.421249493Z"}  
{"log":"org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device\n","stream":"stdout","time":"2017-09-10T09:23:45.421269749Z"}  
...

{"log":"[2017-09-10T13:51:21,330][WARN][o.e.c.r.a.DiskThresholdMonitor] [ywpbCov] high disk watermark [90%] exceeded on [ywpbCovzSIGDPk8wiIG2IA][ywpbCov][/usr/share/elasticsearch/data/nodes/0] free: 20kb[1.9E-5%], shards will be relocated away from this node\n","stream":"stdout","time":"2017-09-10T13:51:21.331746084Z"}

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [September 11, 2017, 3:51pm UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/10 "2017-09-11T15:51:51Z")

</div>

It would seem like Elasticsearch has run out of disk space. You will need to add more or scale out the cluster.

---

<div class="post-metadata">

### Author: ![rchumarin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rchumarin/32/23456_2.png) [@rchumarin](https://discuss.elastic.co/u/rchumarin)
#### Post date: [September 12, 2017, 7:13am UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/11 "2017-09-12T07:13:23Z")

</div>

Thank you

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 10, 2017, 7:13am UTC](https://discuss.elastic.co/t/logs-do-not-displayed-in-elk-what-is-the-problem-how-to-fix-the-error/100058/12 "2017-10-10T07:13:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
