Logs do not displayed in ELK. What is the problem? How to fix the error?

Elastic Stack Elasticsearch
1

Hello.

Kibana version: 5.4.2
Elasticsearch version: 5.4.2
Logstash version: 5.4.2
Server OS version: Linux Red Hat, docker container kibana, logstash, elasticsearch and curator.
Logs do not displayed in ELK. What is the problem? How to fix the error?

Discover: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
Less

• Error: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
• at http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10760
• at Function.Promise.try (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:22203)
• at http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21573
• at Array.map (native)
• at Function.Promise.map (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21528)
• at callResponseHandlers (http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10376)
• at http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:27:29944
• at processQueue (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23621)
• at http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23888
• at Scope.$eval (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:39:4619)

2

It seems to be a problem with Elasticsearch. Is there anything in the Elasticsearch logs?

3

We perform load testing of the java application. From java application should be logs in the stream stdout. The docker log-driver gelf must send them to logstash.
But there are no logs in elasticsearch journals.

http://10.118.41.32:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana NrnfhK-_TIKoow0beow72w 1 1 2 0 10.6kb 10.6kb
red open messenger-2017.09.10 3AHvYolPQraNW-eGazlZZQ 5 1

4

Have you looked for logs in the log directory Elasticsearch has configured?

It looks like you have a problem with the messenger-2017.09.10 index as it is in red state. The logs should hold more details.

5

I'm looking for logs through kibana. There are no logs in the kibana

6

Elasticsearch logs are not available through Kibana, so you need to look in the file system.

7

Tell me please, in what elasticsearch file are the logs stored?

8

As you are using docker, they might be accessible via docker logs.

9

{"log":"[2017-09-10T09:23:45,416][WARN ][o.e.i.e.Engine ] [ywpbCov] [messenger-2017.09.10][1] failed engine [merge failed]\n","stream":"stdout","time":"2017-09-10T09:23:45.421249493Z"}
{"log":"org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device\n","stream":"stdout","time":"2017-09-10T09:23:45.421269749Z"}
...

{"log":"[2017-09-10T13:51:21,330][WARN ][o.e.c.r.a.DiskThresholdMonitor] [ywpbCov] high disk watermark [90%] exceeded on [ywpbCovzSIGDPk8wiIG2IA][ywpbCov][/usr/share/elasticsearch/data/nodes/0] free: 20kb[1.9E-5%], shards will be relocated away from this node\n","stream":"stdout","time":"2017-09-10T13:51:21.331746084Z"}

10

It would seem like Elasticsearch has run out of disk space. You will need to add more or scale out the cluster.

11

Thank you

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.