Curl: (7) Failed to connect to 10.x.x.x port 5601: Connection refused

Here, I provide the Procedure what I follow to setup a Elastic Search and Kibana on GCP Vm's:

  1. Created a 2 Vm's of Es and Kibana with Reserved Internal and External Ip's.
  2. Successfully Deployed Es and Kibana Debian packages On respective Vm's.
  3. Successfully Reconfigured the YAML files for both ES and Kibana i.e;
    elasticsearch.yml:
node.name: elasticsearch
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: Internal_ip addr of elasticseach
http.port: 9200

Kibana.yml:

server.port: 5601
server.host: "Internal_IP addr of Kibana"
server.publicBaseUrl: "http://Internal_IP addr Of Kibana:5601"
elasticsearch.hosts: ["https://Internal_IP Addr of Elasticsearch:9200"]
elasticsearch.username: "elastic"
elasticsearch.password: "xxxxxxxx"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/http_ca.crt" ]

4.Successfully Set the required ports(9200 and 5601) on Respectve Vm's.

   firewall-cmd --list-all
public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh dhcpv6-client
  ports: 5601/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: ssh dhcpv6-client
  ports: 9200/tcp 9200/udp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

5. So, I saw the messages of Es and Kibana are active after the starting the services
6. When I connect to my elastic Instance from Terminal
    curl -XGET -u elastic https://Internal_IP addr of elasticsearch:9200 -k
output:
    ```
{
  "name" : "elasticsearch",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "U0_1cpklQ2yitFxJZ5XJTA",
  "version" : {
    "number" : "8.7.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "09520b59b6bc1057340b55750186466ea715e30e",
    "build_date" : "2023-03-27T16:31:09.816451435Z",
    "build_snapshot" : false,
    "lucene_version" : "9.5.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
  1. When I connect to my Kibana Instance from Terminal
curl -XGET "https://Internal_Ip Addr of Kibana:5601"
Output:
  curl: (7) Failed to connect to 10.x.x.x port 5601: Connection refused

So can anyone help me to solve

Hi @ram_222 Welcome to the community! I formatted your post because it was very difficult to read. Please try to format in the future.

Kibana is most likely running on http not https according to your configurations.

@stephenb I tried with http but it throws same error

curl -XGET "http://10.x.x.x:5601"
curl: (7) Failed to connect to 10.x.x.x port 5601: Connection refused

try

curl-v -XGET "http://10.x.x.x:5601"

Looks like a connectivity issue... OR kibana is not running at all.

How did you start Kibana? Did you check the logs?

what is the result of

systemctl status kibana

What do the logs show...
tail -f /var/log/kibana/kibana.log

@stephenb
First Checked the status of Kibana

root@kibana:~# sudo systemctl status kibana.service 
● kibana.service - Kibana
   Loaded: loaded (/usr/lib/systemd/system/kibana.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2023-04-17 15:13:40 UTC; 13s ago
     Docs: https://www.elastic.co
 Main PID: 26257 (node)
    Tasks: 11 (limit: 4660)
   CGroup: /system.slice/kibana.service
           └─26257 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Apr 17 15:13:40 kibana systemd[1]: Started Kibana.
Apr 17 15:13:42 kibana kibana[26257]: [2023-04-17T15:13:42.925+00:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui]

Connecting from Terminal

 curl -v -XGET "http://10.x.x.x:5601"
Output:
Note: Unnecessary use of -X or --request, GET is already inferred.
* Rebuilt URL to: http://10.x.x.x:5601/
*   Trying 10.x.x.x...
* TCP_NODELAY set
* connect to 10.x.x.x port 5601 failed: Connection refused
* Failed to connect to 10.x.x.x port 5601: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.x.x.x port 5601: Connection refused

Here's the Output of Checking the logs:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:10:57.104+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":25602},"trace":{"id":"ed71516e3b454e33ca7c2943b05c6bbd"},"transaction":{"id":"9a008eabb5f34d3c"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:11:17.134+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":25658},"trace":{"id":"823aa52f0746530d51dac6eb7187adbf"},"transaction":{"id":"7d6799f32c1a666a"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:11:37.361+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":25711},"trace":{"id":"7d01ffeacd4b3f5f05a6a7f56405f613"},"transaction":{"id":"4e30b48025155663"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:11:57.333+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":25758},"trace":{"id":"5cf1e068f004f2ee35f49514e586f791"},"transaction":{"id":"3ca70368ac3eea4a"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:13:42.925+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":26257},"trace":{"id":"89e21df34bcf65ee79ae8285f0cea217"},"transaction":{"id":"4d8103b848bd8999"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:14:02.582+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":26305},"trace":{"id":"ce2cde0ad2e90eb7e271f7d012a22e22"},"transaction":{"id":"7775866b70bcdd37"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-17T15:14:22.550+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":26347},"trace":{"id":"bb41fddf1918da9650454e40d2349585"},"transaction":{"id":"61de08ab968615ce"}}

Looks like a connectivity issue to me...

You might need to look back further into the logs but at first look looks like kibana to be running...

do you see kibana listening on 5601

The Kibana Port isn't listening in Kibana Instance

netstat -anpe | grep LISTEN
output:
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        22012      1241/systemd-resolv 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          25384      1529/sshd           
tcp6       0      0 :::22                   :::*                    LISTEN      0          25386      1529/sshd

The Es port is Listening in ES Instance

netstat -anpe | grep LISTEN
output:
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      101        22080      1462/systemd-resolv 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          26056      1762/sshd           
tcp6       0      0 :::9200                 :::*                    LISTEN      112        28816      1938/java           
tcp6       0      0 10.x.x.x:9300        :::*                    LISTEN      112        28476      1938/java           
tcp6       0      0 :::22                   :::*                    LISTEN      0          26058      1762/sshd   

but I applied a Firewall rule to Kibana and Es Instance's to allow all IP Addresses(0.0.0.0/0) with specified ports of 9200,9300,5601

You need to look further at the kibana logs.

Stop Kibana
Remove the Log file.
Start kibana and look at the entire logs

Not sure what the issue is

You can try setting in kibana.yml

server.host: 0.0.0.0

First I tried what you say

Stop Kibana
Remove the Log file.
Start kibana and look at the entire logs

So at that time the logs is be:

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:14:41.967+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6242},"trace":{"id":"47c3c74c1cc67e8523b45abbf4a13b70"},"transaction":{"id":"b6b6876751f5d379"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:15:01.248+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6291},"trace":{"id":"8b0b30466343b8460358b671293465eb"},"transaction":{"id":"b7c28f370febabdb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:15:20.773+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6344},"trace":{"id":"3cda5c8c8c8355f32ae164bfcec5ab7c"},"transaction":{"id":"d474d5708cfcf3e4"}}

After I change server.host: 0.0.0.0 in Kibana.yml file then I restarted the kibana Service and Checked the logs i.e;

{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:14:41.967+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6242},"trace":{"id":"47c3c74c1cc67e8523b45abbf4a13b70"},"transaction":{"id":"b6b6876751f5d379"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:15:01.248+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6291},"trace":{"id":"8b0b30466343b8460358b671293465eb"},"transaction":{"id":"b7c28f370febabdb"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:15:20.773+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6344},"trace":{"id":"3cda5c8c8c8355f32ae164bfcec5ab7c"},"transaction":{"id":"d474d5708cfcf3e4"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:17:16.619+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6407},"trace":{"id":"80a0445fcbe7e733cfec1507198eb0f8"},"transaction":{"id":"c60eb58f836b5692"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:17:36.280+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6456},"trace":{"id":"eb57e29bf1035ff462d50a7270d126f9"},"transaction":{"id":"2f51235420d389c3"}}
{"service":{"node":{"roles":["background_tasks","ui"]}},"ecs":{"version":"8.6.0"},"@timestamp":"2023-04-18T05:17:56.248+00:00","message":"Kibana process configured with roles: [background_tasks, ui]","log":{"level":"INFO","logger":"node"},"process":{"pid":6508},"trace":{"id":"0a4b822d644391b5c91b2c64cc2593c6"},"transaction":{"id":"0c0a3dcf3f6d31a0"}}

Then I try to connect from my terminal but same error gives

curl -v -XGET "http://10.x.x.x:5601"
Note: Unnecessary use of -X or --request, GET is already inferred.
* Rebuilt URL to: http://10.x.x.x:5601/
*   Trying 10.x.x.x...
* TCP_NODELAY set
* connect to 10.x.x.x port 5601 failed: Connection refused
* Failed to connect to 10.x.x.x port 5601: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 10.x.x.x port 5601: Connection refused

Hello @stephenb....can you please look at the issue

Please be patient There are many questions on this forum and yours is no more important than anyone else's.

There's no SLA. This is a community forum and we are not all in the same time zone.

I do not know what's happening.

There's something basic going on. You can't see the port listening.

The logs that you provided are not useful There are many other logs in the Kibana.

You can set the log level to debug.

You could try downloading the tar Ball of Kibana, set your same settings and started in the foreground and watch the logs.

There is something basic going on.

Usually in these cases, even though you check the firewalls there is some connectivity issue permission issue, something like that something with your local environment.

After changing the default network to Custom VPC(Virtual Private Cloud) for an instance and also changed some configuration settings in YAML files.

So, successfully browse the Kibana.

Thanks for your support

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.