Logs do not displayed in ELK. What is the problem? How to fix the error?


(Chumarin Rafael) #1

Hello.

Kibana version: 5.4.2
Elasticsearch version: 5.4.2
Logstash version: 5.4.2
Server OS version: Linux Red Hat, docker container kibana, logstash, elasticsearch and curator.
Logs do not displayed in ELK. What is the problem? How to fix the error?


Discover: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
Less

• Error: Request to Elasticsearch failed: {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
• at http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10760
• at Function.Promise.try (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:22203)
• at http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21573
• at Array.map (native)
• at Function.Promise.map (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:82:21528)
• at callResponseHandlers (http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:28:10376)
• at http://10.118.41.32:5601/bundles/kibana.bundle.js?v=15117:27:29944
• at processQueue (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23621)
• at http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:38:23888
• at Scope.$eval (http://10.118.41.32:5601/bundles/commons.bundle.js?v=15117:39:4619)


(Christian Dahlqvist) #2

It seems to be a problem with Elasticsearch. Is there anything in the Elasticsearch logs?


(Chumarin Rafael) #3

We perform load testing of the java application. From java application should be logs in the stream stdout. The docker log-driver gelf must send them to logstash.
But there are no logs in elasticsearch journals.


http://10.118.41.32:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana NrnfhK-_TIKoow0beow72w 1 1 2 0 10.6kb 10.6kb
red open messenger-2017.09.10 3AHvYolPQraNW-eGazlZZQ 5 1


(Christian Dahlqvist) #4

Have you looked for logs in the log directory Elasticsearch has configured?

It looks like you have a problem with the messenger-2017.09.10 index as it is in red state. The logs should hold more details.


(Chumarin Rafael) #5

I'm looking for logs through kibana. There are no logs in the kibana


(Christian Dahlqvist) #6

Elasticsearch logs are not available through Kibana, so you need to look in the file system.


(Chumarin Rafael) #7

Tell me please, in what elasticsearch file are the logs stored?


(Christian Dahlqvist) #8

As you are using docker, they might be accessible via docker logs.


(Chumarin Rafael) #9

{"log":"[2017-09-10T09:23:45,416][WARN ][o.e.i.e.Engine ] [ywpbCov] [messenger-2017.09.10][1] failed engine [merge failed]\n","stream":"stdout","time":"2017-09-10T09:23:45.421249493Z"}
{"log":"org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device\n","stream":"stdout","time":"2017-09-10T09:23:45.421269749Z"}
...

{"log":"[2017-09-10T13:51:21,330][WARN ][o.e.c.r.a.DiskThresholdMonitor] [ywpbCov] high disk watermark [90%] exceeded on [ywpbCovzSIGDPk8wiIG2IA][ywpbCov][/usr/share/elasticsearch/data/nodes/0] free: 20kb[1.9E-5%], shards will be relocated away from this node\n","stream":"stdout","time":"2017-09-10T13:51:21.331746084Z"}


(Christian Dahlqvist) #10

It would seem like Elasticsearch has run out of disk space. You will need to add more or scale out the cluster.


(Chumarin Rafael) #11

Thank you


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.