Unable to revive connection: http://localhost:9200

pyerunka · November 19, 2019, 5:54am

Hello,

Can anyone help me in this??
Currently i have configured ES cluster. After that when i was trying to start kibana, it is giving me below mentioned error:

log [05:01:29.789] [warning][admin][elasticsearch] Unable to revive connection: http://localhost:9200/

I am working on ES version 7.4.2.

Regards,
Priyanka

flash1293 · November 19, 2019, 9:59am

Hi @pyerunka,

To find the culprit a bit more information about your setup will be helpful.

Are you running Elasticsearch and Kibana on the same host?
Are you using docker?
Are you reaching Elasticsearch on the same host you are running Kibana on when using curl: curl -v http://localhost:9200?
Could you copy/paste your complete startup log?

pyerunka · November 19, 2019, 10:18am

Hello @flash1293,

Thanks for your reply!!!

Yes, i am running ES and kibana on same host.
No i am not using docker.
3.Yes i can reach to ES on the same host where i am running kibana.
Please find below startup log:

log [10:05:38.802] [info][plugins-system] Setting up [4] plugins: [security,inspector,data,translations]
log [10:05:38.810] [info][plugins][security] Setting up plugin
log [10:05:38.813] [warning][config][plugins][security] Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidatedon restart, please set xpack.security.encryptionKey in kibana.yml
log [10:05:38.814] [warning][config][plugins][security] Session cookies willbe transmitted over insecure connections. This is not recommended.
log [10:05:38.909] [info][data][plugins] Setting up plugin
log [10:05:38.910] [info][plugins][translations] Setting up plugin
log [10:05:38.912] [info][plugins-system] Starting [3] plugins: [security,data,translations]
log [10:06:16.154] [info][status][plugin:kibana@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.161] [info][status][plugin:elasticsearch@7.4.2] Status changedfrom uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.164] [info][status][plugin:xpack_main@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.174] [info][status][plugin:telemetry@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.176] [info][status][plugin:graph@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.187] [info][status][plugin:monitoring@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.192] [info][status][plugin:spaces@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.227] [info][status][plugin:security@7.4.2] Status changed fromuninitialized to green - Ready
log [10:06:16.230] [info][status][plugin:searchprofiler@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.234] [info][status][plugin:ml@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.285] [info][status][plugin:tilemap@7.4.2] Status changed fromuninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.288] [info][status][plugin:watcher@7.4.2] Status changed fromuninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.300] [info][status][plugin:grokdebugger@7.4.2] Status changedfrom uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.307] [info][status][plugin:dashboard_mode@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.383] [info][status][plugin:interpreter@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.386] [info][status][plugin:tile_map@7.4.2] Status changed fromuninitialized to green - Ready
log [10:06:16.390] [info][status][plugin:task_manager@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.394] [info][status][plugin:maps@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.415] [info][status][plugin:canvas@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.421] [info][status][plugin:license_management@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.424] [info][status][plugin:cloud@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.428] [info][status][plugin:index_management@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.444] [info][status][plugin:console@7.4.2] Status changed fromuninitialized to green - Ready
log [10:06:16.451] [info][status][plugin:index_lifecycle_management@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.464] [info][status][plugin:metrics@7.4.2] Status changed fromuninitialized to green - Ready
log [10:06:16.501] [info][status][plugin:infra@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.505] [info][status][plugin:rollup@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.514] [info][siem] Plugin initializing
log [10:06:16.544] [info][siem] Plugin done initializing
log [10:06:16.547] [info][status][plugin:siem@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.553] [info][status][plugin:remote_clusters@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.559] [info][status][plugin:cross_cluster_replication@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.581] [info][status][plugin:upgrade_assistant@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.617] [info][status][plugin:uptime@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.622] [info][status][plugin:oss_telemetry@7.4.2] Status changedfrom uninitialized to green - Ready
log [10:06:16.626] [info][status][plugin:file_upload@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.631] [warning][encrypted_saved_objects] Generating a random key for xpack.encrypted_saved_objects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encrypted_saved_objects.encryptionKey in kibana.yml
log [10:06:16.634] [info][status][plugin:encrypted_saved_objects@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:16.650] [info][status][plugin:snapshot_restore@7.4.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [10:06:16.664] [info][status][plugin:kibana_react@7.4.2] Status changedfrom uninitialized to green - Ready
log [10:06:16.670] [info][status][plugin:region_map@7.4.2] Status changed from uninitialized to green - Ready
log [10:06:17.990] [error][admin][elasticsearch] Request error, retrying HEAD http://localhost:9200/.apm-agent-configuration => connect ECONNREFUSED 127.0.0.1:9200
log [10:06:18.090] [error][admin][elasticsearch] Request error, retrying GET http://localhost:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => connect ECONNREFUSED 127.0.0.1:9200
log [10:06:19.183] [warning][admin][elasticsearch] Unable to revive connection: http://localhost:9200/
log [10:06:19.185] [warning][admin][elasticsearch] No living connections Could not create APM Agent configuration: No Living connections
log [10:06:19.188] [warning][admin][elasticsearch] Unable to revive connection: http://localhost:9200/
log [10:06:19.189] [warning][admin][elasticsearch] No living connections

Regards,
Priyanka

flash1293 · November 19, 2019, 10:23am

Seems like Elasticsearch is not picking up the connection Kibana tries to initiate. Could you run curl -v http://localhost:9200and post the response?

Besides that could you provide your kibana.yml and elasticsearch.yml?

pyerunka · November 19, 2019, 10:36am

Hello @flash1293,

Thanks for reply!!!
I tried for curl -v http://localhost:9200

Rebuilt URL to: http://localhost:9200/
Hostname was NOT found in DNS cache
Trying ... (I am hiding Ip address)...
Connected to Localhost (...) port 9200
GET / HTTP/1.1
User-Agent: curl/7.37.0
Host: Localhost:9200
Accept: /
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-length: 530

"name" : "Master-node",
"cluster_name" : "InTouch",
"cluster_uuid" : "gzXXocd1T7WVqeNeZrwLKw",
"version" : {
"number" : "7.4.2",
"build_flavor" : "default",
"build_type" : "zip",
"build_hash" : "2f90bbf7b93631e52bafb59b3b049cb44ec25
"build_date" : "2019-10-28T20:40:44.881551Z",
"build_snapshot" : false,
"lucene_version" : "8.2.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"

And i have not updated anything in kibana.yml file. Means that file is as it is when we download kibana from Elastic.co.
Please find Elasticsearch.yml file:

cluster.name: InTouch
cluster.remote.connect: false
network.host: abcd.com
transport.host: abcd.com
transport.tcp.port: 9300-9400
node.name: Master-node
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: false
node.ingest: true
discovery.zen.ping.unicast.hosts: [abcd.com, xyz.com]
cluster.initial_master_nodes: ["Master-node"]
bootstrap.memory_lock: true

Regards,
Priyanka

flash1293 · November 19, 2019, 10:55am

Hm, maybe the network.host: abcd.com setting prevents Elasticsearch to bind the local loopback address. In the debug output from curl you see that curl is translating localhost into something else than 127.0.0.1, but Kibana apparently can't do this lookup (probably due to host aliases or something like that).

There are probably two ways to resolve this

either use the same host name for your elasticsearch cluster as curl or other consumers (the ip curl used or abcd.com)
or explicitly also bind Elasticsearch to the loopback interface in addition to the public one. You should be able to do this by setting network.bind_host to _local_ and abcd.com and network.publish_host to abcd.com instead of setting network.host https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings

Let me know whether that helps

pyerunka · November 19, 2019, 11:00am

Hello @flash1293,

I am purposely changed network.host to abcd.com. I do not want to showcase it as security concerns.
I am using host name same as elasticsearch cluster as curl.

Regards,
Priyanka

flash1293 · November 19, 2019, 11:04am

All right, let me know when you run into further problems.

pyerunka · November 20, 2019, 9:24am

Hello @flash1293,

Thanks for your great help!!!
My issue has been resolved. I am able to start kibana.
I changed network.bind_host to _local_ and network.publish_host to abcd.com instead of setting network.hostas mentioned in above comment.

Regards,
Priyanka

pyerunka · November 21, 2019, 10:02am

Hello @flash1293,

This issue is resolved on my master node.
I am not able to start kibana on my data node also. I am getting same error as i have mentioned in earlier comments.

Can you help me in this?

Regards,
Priyanka

flash1293 · December 3, 2019, 11:13am

You can try to add _site_ instead of _local_ to the list of hosts. Then hosts in the same local network should also be able to connect

system · December 31, 2019, 11:26am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.