Connection refused error from logstash.log

Elastic Stack Logstash
1

I am getting following error on elk server and not able to see the log on Kibana.

{:timestamp=>"2016-10-28T21:52:02.243000+0000", :message=>"Connection refused", :class=>"Manticore::SocketException",
{:timestamp=>"2016-10-28T21:52:07.129000+0000", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["http://elk.marathon.mesos:9200"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused", :class=>"Manticore::SocketException", :level=>:error}

But something weird is happening , I can see translog and index files have been created in elasticsearch under
/var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2016.10.28/0/ folder.

Can someone help me why I can't visualize the log on Kibana.?
even I tried to curl on "elk.marathon.mesos:9200" but got only following JSON
{
"name" : "Airborne",
"cluster_name" : "elasticsearch",
"version" : {
"number" : "2.4.0",
"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
"build_timestamp" : "2016-08-29T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}

2

{:timestamp=>"2016-10-28T21:52:02.243000+0000", :message=>"Connection refused", :class=>"Manticore::SocketException",
{:timestamp=>"2016-10-28T21:52:07.129000+0000", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["http://elk.marathon.mesos:9200"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused", :class=>"Manticore::SocketException", :level=>:error}

But something weird is happening , I can see translog and index files have been created in elasticsearch under
/var/lib/elasticsearch/elasticsearch/nodes/0/indices/filebeat-2016.10.28/0/ folder.

Maybe ES was temporarily unavailable?

Can someone help me why I can't visualize the log on Kibana.?
even I tried to curl on "elk.marathon.mesos:9200" but got only following JSON

What did you expect would've happened?

3

I should see the logs on Kibana which I didn't and when I did curl elastic search, should see all logs not jut default message.

4

I don't think it's temporarily unavailable issue as tried few times to stop and start elk server but got same error
each time.

5

I should see the logs on Kibana which I didn't and when I did curl Elasticsearch, should see all logs not jut default message.

If you want to use curl to view what's stored in ES you'll have to use one of the query APIs. The root resource (http://somehostname:9200) will never return any documents.

I don't think it's temporarily unavailable issue as tried few times to stop and start elk server but got same error each time.

Assuming ES is actually starting up it sounds like a more permanent network-related problem. When you're successfully running curl, are you doing it from the Logstash host?

6

Thank you so much for your response.
Your are right. After using the elasticserach api like curl "http://###.###.###.###:9200/_search?size=10&pretty=true", I am able to see the logs.

The only issue I have is that Kibana is not displaying any logs.
I have a single image of elk stack and deployed in a single container. So running the curl on same Logstash host.
Got following response after curl the kibana

I don't know what else I can debug as logs are flowing from filebeat to logstash and logstash to elastic search.

Thanks
Manoj

7

Finally able to find the root cause of this issue which was index name . By default Kibana use logstash-* but I have used filebeat-*.So after creating a new index pattern with name [filebeat-]YYYY.MM.DD, able to see the logs on Kibana. Always thank you for your help.