Unable to connect to Kibana using an IP address


#1

I have installed Kibana and Elastic search on one of my organization's server. The server runs on Cent OS 6.6.

I have been following this tutorial [Link](https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7)

When I try pointing Kibana to the browser, I am unable to reach the site. When I point to the browser URL

http://10.1.151.28:5601

I get an error like this , I get ERR_CONNECTION_TIMED_OUT.

Here is the kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
# server.port: 5601

# This setting specifies the IP address of the back end server.
  server.host: "10.1.151.28"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This setting
# cannot end in a slash.
# server.basePath: ""

# The maximum payload size in bytes for incoming server requests.
# server.maxPayloadBytes: 1048576

# The URL of the Elasticsearch instance to use for all your queries.
# elasticsearch.url: "http://localhost:9200"

# When this setting’s value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
# elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn’t already exist.
# kibana.index: ".kibana"

# The default application to load.
# kibana.defaultAppId: "discover"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
# elasticsearch.username: "user"
# elasticsearch.password: "pass"

# Paths to the PEM-format SSL certificate and SSL key files, respectively. These
# files enable SSL for outgoing requests from the Kibana server to the browser.
# server.ssl.cert: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
# elasticsearch.ssl.cert: /path/to/your/client.crt
# elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
# elasticsearch.ssl.ca: /path/to/your/CA.pem

# To disregard the validity of SSL certificates, change this setting’s value to false.
# elasticsearch.ssl.verify: true

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
# elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
# elasticsearch.requestTimeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
# elasticsearch.shardTimeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
# elasticsearch.startupTimeout: 5000

# Specifies the path where Kibana creates the process ID file.
# pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
# logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
# logging.silent: false

I checked Kibana's status using

sudo service kibana status

Kibana is running successfully.

Is there anything wrong I am doing? Does anything needs to be changed?


Unable to connect to Kibana on public IP
(kulkarni) #2

Has firewall/iptables been configured to allow traffic on the 5601 port? What happens when you directly ping the URL via CUrl ...something like `curl -XGET 'http://localhost:5601/' to the server ..what do you get ?

any more additional logs?

Thanks
Rashmi


#3

Hi Rashmi,

when I do

curl -XGET http://10.1.151.28:5601

I get, curl: (7) couldn't connect to host


#4

Also, how do I configure firewall/iptables to allow traffic on the port 5601?


(kulkarni) #5

Can you please do iptables -F and see if it works. ( This will flush all other firewall rules - do not use on Production)
Also please do `curl -XGET 'http://localhost:5601/' on the server itself and see if the ping goes through ?
Let us know

Thanks
Rashmi


#6

When I do curl -XGET http://localhost:5601/, I get

<script>var hashRoute = '/app/kibana';
var defaultRoute = '/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>

I did iptables -F and it seemed like it worked.

Later, I executed the command sudo iptables -L.

I got this output,

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Just wanted to let you know, I have installed Kibana on a remote server running on Cent OS. I am accessing the server on Windows using Putty. So, I am trying to access the browser URL on my windows machine. It should be fine right?


(kulkarni) #7

Yes, as long as the 2 machines are able to ping each other , you should be fine . You could try

curl http://localhost:5601/api/status

{"name":"ubuntu","uuid":"5b2de169-2785-441b-ae8c-186a1936b17d","version":{"number":"7.0.0-alpha1","build_hash":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX","build_number":9007199254740991,"build_snapshot":false},"status":{"overall":{"state":"green","title":"Green","nickname":"Looking good","icon":"success","since":"2018-07-02T21:47:23.199Z"},"statuses":[{"id":"plugin:kibana@7.0.0-alpha1","state":"green","icon":"success","message":"Ready","since":"2018-07-02T21:47:23.199Z"},{"id":"plugin:elasticsearch@7.0.0-alpha1","state":"green","icon":"success","message":"Ready","since":"2018-07-02T21:47:24.368Z"},{"id":"plugin:timelion@7.0.0-alpha1","state":"green","icon":"success","message":"Ready","since":"2018-07-02T21:47:24.224Z"},{"id":"plugin:console@7.0.0-alpha1","state":"green","icon":"success","message":"Ready","since":"2018-07-02T21:47:24.228Z"},{"id":"plugin:metrics@7.0.0-alpha1","state":"green","icon":"success","message":"Ready","since":"2018-07-02T21:47:24.230Z"}]},"metrics":{"last_updated":"2018-07-02T21:47:54.256Z","collection_interval_in_millis":5000,"uptime_in_millis":85126,"process":{"mem":{"heap_max_in_bytes":319778816,"heap_used_in_bytes":232098088,"resident_set_size_in_bytes":376672256,"external_in_bytes":1033928},"pid":3582,"uptime_ms":85126},"os":{"cpu":{"load_average":{"1m":1.46337890625,"5m":2.43896484375,"15m":2.9921875}},"mem":{"free_in_bytes":5462851584,"total_in_bytes":33548386304},"cgroup":{"cpuacct":{"control_group":"/user.slice","usage_nanos":372923421692918},"cpu":{"control_group":"/user.slice","cfs_period_micros":100000,"cfs_quota_micros":-1,"stat":{"number_of_elapsed_periods":0,"number_of_times_throttled":0,"time_throttled_nanos":0}}}},"response_times":{"max_in_millis":0},"requests":{"total":0,"disconnects":0,"status_codes":{}},"concurrent_connections":0,"sockets":{"http":{"total":0},"https":{"total":0}},"event_loop_delay":0.8120279908180237}}

#8

This is the output I get when I run the above command

{"status":{"overall":{"state":"green","title":"Green","nickname":"Looking good","icon":"success","since":"2018-06-29T23:54:58.935Z"},"statuses":[{"name":"plugin:kibana","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:58.935Z"},{"name":"plugin:elasticsearch","state":"green","icon":"success","message":"Kibana index ready","since":"2018-06-29T23:55:08.172Z"},{"name":"plugin:kbn_vislib_vis_types","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:58.998Z"},{"name":"plugin:markdown_vis","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:59.016Z"},{"name":"plugin:metric_vis","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:59.023Z"},{"name":"plugin:spyModes","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:59.029Z"},{"name":"plugin:statusPage","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:59.042Z"},{"name":"plugin:table_vis","state":"green","icon":"success","message":"Ready","since":"2018-06-29T23:54:59.053Z"}]},"metrics":{"heapTotal":[[1530569771252,376102240],[1530569766251,376102240],[1530569761249,376102240],[1530569756247,376102240],[1530569751245,375070304],[1530569746243,375070304],[1530569741242,375070304],[1530569736241,375070304],[1530569731240,375070304],[1530569726237,375070304],[1530569721236,375070304],[1530569716233,375070304]],"heapUsed":[[1530569771252,368535664],[1530569766251,368803664],[1530569761249,368507096],[1530569756247,368208344],[1530569751245,368471360],[1530569746243,368173656],[1530569741242,367875712],[1530569736241,368138160],[1530569731240,367827824],[1530569726237,368087968],[1530569721236,367790856],[1530569716233,367491864]],"load":[[1530569771252,[0,0,0]],[1530569766251,[0,0,0]],[1530569761249,[0,0,0]],[1530569756247,[0,0,0]],[1530569751245,[0,0,0]],[1530569746243,[0,0,0]],[1530569741242,[0,0,0]],[1530569736241,[0,0,0]],[1530569731240,[0,0,0]],[1530569726237,[0,0,0]],[1530569721236,[0,0,0]],[1530569716233,[0,0,0]]],"responseTimeAvg":[[1530569771252,null],[1530569766251,null],[1530569761249,null],[1530569756247,null],[1530569751245,null],[1530569746243,null],[1530569741242,null],[1530569736241,null],[1530569731240,null],[1530569726237,null],[1530569721236,null],[1530569716233,null]],"responseTimeMax":[[1530569771252,0],[1530569766251,0],[1530569761249,0],[1530569756247,0],[1530569751245,0],[1530569746243,0],[1530569741242,0],[1530569736241,0],[1530569731240,0],[1530569726237,0],[1530569721236,0],[1530569716233,0]],"requestsPerSecond":[[1530569771252,0],[1530569766251,0],[1530569761249,0],[1530569756247,0],[1530569751245,0],[1530569746243,0],[1530569741242,0],[1530569736241,0],[1530569731240,0],[1530569726237,0],[1530569721236,0],[1530569716233,0]]}}


#9

So, what do I do now? Is the output that I got after running curl http://localhost:5601/api/status, correct?


(kulkarni) #10

localhost -api status response seems right . But per your initial q, you would like the IP in the URL right ? Have you tried restarting the server after making changes? That would be my guess . Restart should take in your changes.

Cheers
Rashmi


#11

Hi Rashmi,

When I run curl http://localhost:5601/api/status, it says failed to connect. But, replacing localhost with the public IP works.

Here is my updated elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
# cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: node-1
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.memory_lock: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: 10.1.151.30
#
# Set a custom port for HTTP:
#
 http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true

#12

Hi Rashmi,

I am still unable to open Kibana. When I run curl http://localhost:5601/api/status, it says failed to connect. But, replacing localhost with the public IP works.

Here is my updated elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please see the documentation for further information on configuration options:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
# cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: node-1
#
# Add custom attributes to the node:
#
# node.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
# path.data: /path/to/data
#
# Path to log files:
#
# path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
# bootstrap.memory_lock: true
#
# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
# available on the system and that the owner of the process is allowed to use this limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
 network.host: 10.1.151.30
#
# Set a custom port for HTTP:
#
 http.port: 9200
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
# discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
#
# discovery.zen.minimum_master_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
# gateway.recover_after_nodes: 3
#
# For more information, see the documentation at:
# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
#
# ---------------------------------- Various -----------------------------------
#
# Disable starting multiple nodes on a single system:
#
# node.max_local_storage_nodes: 1
#
# Require explicit names when deleting indices:
#
# action.destructive_requires_name: true

#13

Here is the updated kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# This setting specifies the IP address of the back end server.
server.host: "10.1.151.30"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This setting
# cannot end in a slash.
# server.basePath: ""

# The maximum payload size in bytes for incoming server requests.
# server.maxPayloadBytes: 1048576

# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://10.1.151.30:9200"

# When this setting’s value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
# elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn’t already exist.
# kibana.index: ".kibana"

# The default application to load.
# kibana.defaultAppId: "discover"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
# elasticsearch.username: "user"
# elasticsearch.password: "pass"

# Paths to the PEM-format SSL certificate and SSL key files, respectively. These
# files enable SSL for outgoing requests from the Kibana server to the browser.
# server.ssl.cert: /path/to/your/server.crt
# server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files validate that your Elasticsearch backend uses the same key files.
# elasticsearch.ssl.cert: /path/to/your/client.crt
# elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
# elasticsearch.ssl.ca: /path/to/your/CA.pem

# To disregard the validity of SSL certificates, change this setting’s value to false.
# elasticsearch.ssl.verify: true

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
# elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
# elasticsearch.requestTimeout: 300000

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
# elasticsearch.shardTimeout: 0

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
# elasticsearch.startupTimeout: 5000

# Specifies the path where Kibana creates the process ID file.
# pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
# logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
# logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
# logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
# logging.verbose: false

Here is the updated kibana.conf (/etc/nginx/conf.d/kibana.conf) file:

server {
    listen 80;

    server_name 10.1.151.30:5601;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://10.1.151.30:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

Here is the nginx.conf (/etc/nginx/nginx.conf) file.

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;
}

Also here is a pic of the java version I am running,

Java

What wrong am I doing. Are the config files correct?


(Bill McConaghy) #14

This config all looks correct. Can you confirm that http://<public ip>/api/status works on a remote machine (not the machine that Kibana is running on)?


#15

Hi Bill,

I tried opening the url from the Chrome browser on my Windows machine. I entered in the browser URL

http://10.1.151.30/api/status.

I got a connection timed out request

ERR_CONNECTION_TIMED_OUT

What am I supposed to do now?


#16

Would I have to change the firewall settings or install any additional software?


(system) #17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.