Unable to load Kibana Dashboards via MetricBeat

(mark) #1

Hi,

I am running Metricbeat6.5.4 in a Docker container on DCOS. I have been able to send all "Metrics" logs to Elastic search and then to Kibana (v6) successfully.

The issue i am having is when setting the "setup.dashboards.enabled: true" i receive the following error message.

My metricbeat.yml is as follows : "

ERROR	instance/beat.go:800	Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://10.128.0.6:5601/api/status: dial tcp 10.128.0.6:5601: connect: connection refused. Response: .
Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://10.128.0.6:5601/api/status: dial tcp 10.128.0.6:5601: connect: connection refused. Response: ."
###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
fields:
  log_type: metrics


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true
setup.dashboards.file: /etc/metricbeat-6.5.4-linux-x86_64/kibana/6
setup.dashboards.index: "dev_sit_uat-dcos-filebeat*"
#setup.dashboards.always_kibana: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana.host: "10.128.0.6:5601"

  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.128.0.5"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Any suggestions would be greatly appreciated.

(Mario Castro) #2

Hi @mark.penner :slight_smile:

As far as I can see, you are probably habing a network configuration issue and you are not reaching Kibana. Also, if you don't mind, tell us the version of Kibana and Elasticsearch you are using. Are you also installing the stack using the DCOS installer?

You said that you were using DCOS so I'd double check the IP assigned by Marathon as well as the port, specially the port as the dynamic port mapping that Marathon makes me think that it's unlikely that you'll get Kibana default port (maybe yes, if it's fixed).

Also, DCOS uses dynamic host and port resolution so maybe try to see the description of the DCOS service, maybe you need to use KIBANA_HOST and KIBANA_PORT in your configuration for host and port resolution.

(mark) #3

Hi,

thanks for the reply. As requested "Kibana version = 6" "Elasticsearch version=6.5.1"

  1. Is there any specific config in Kibana which i would have to allow ingress traffic from Metricbeat?
    2.There isn't anything specifically set up to allow/disallow network connections. One thing worth noting here is that communication with Logstash is working exactly as intended - so I'm not convinced it's a DC/OS networking problem?

below is the network config i have in DCOS

Network

NETWORK TYPE

HOST

Service Endpoints

NAME PROTOCOL HOST PORT LOAD BALANCED ADDRESS
default tcp 10112 Not Enabled

"portDefinitions": [
{
"port": 10112,
"protocol": "tcp",
"name": "default"
}

is there anything i'm missing from DCOS i require amending? to me this is leaning more towards a Kibana setting i'm missing. If it was a DC/OS networking issue, I would expect all outbound communication to fail. However, the Metricbeats container is exporting its actual metric logs to our Logstash server with no problem.

The Logstash server (10.128.0.5) is on the same subnet as the Metricbeats containers, and the Kibana server (10.128.0.6)"

thanks again.

(Mario Castro) #4

In your Metricbeat configuration file you have this. Have you tried to set it up with setup.kibana.host: "10.128.0.6:10112"?

(mark) #5

Hi yes,

i still receive the following

2019-02-04T14:28:46.936Z INFO instance/beat.go:592 Home path: [/etc/metricbeat-6.5.4-linux-x86_64] Config path: [/etc/metricbeat-6.5.4-linux-x86_64] Data path: [/etc/metricbeat-6.5.4-linux-x86_64/data] Logs path: [/etc/metricbeat-6.5.4-linux-x86_64/logs]
2019-02-04T14:28:46.937Z INFO instance/beat.go:599 Beat UUID: 6f222353-92b1-45f0-9563-2c632437ffaf
2019-02-04T14:28:46.938Z INFO [seccomp] seccomp/seccomp.go:116 Syscall filter successfully installed
2019-02-04T14:28:46.938Z INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "/etc/metricbeat-6.5.4-linux-x86_64", "data": "/etc/metricbeat-6.5.4-linux-x86_64/data", "home": "/etc/metricbeat-6.5.4-linux-x86_64", "logs": "/etc/metricbeat-6.5.4-linux-x86_64/logs"}, "type": "metricbeat", "uuid": "6f222353-92b1-45f0-9563-2c632437ffaf"}}}
2019-02-04T14:28:46.938Z INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:31:39.000Z", "version": "6.5.4"}}}
2019-02-04T14:28:46.938Z INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.6"}}}
2019-02-04T14:28:46.942Z INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07-04T21:31:38Z","containerized":true,"name":"dcos-agentpool0-34294767-000006","ip":["127.0.0.1/8","::1/128","10.0.0.8/16","fe80::20d:3aff:fe0c:a1b8/64","fe80::7c0a:36ff:fec2:4a50/64","198.51.100.1/32","198.51.100.2/32","198.51.100.3/32","fe80::98f6:f7ff:feef:a211/64","fe80::ccd:d9ff:fe2c:af96/64","fe80::72b3:d5ff:fe80:5/64","172.17.0.1/16","fe80::42:22ff:fea5:144a/64","fe80::3ca3:5dff:fe9d:1cd8/64","fe80::88a5:b0ff:fe72:9ca/64","fe80::84c6:98ff:fec2:fffd/64","fe80::681c:8fff:fe71:9513/64","fe80::5c09:c0ff:fea6:3aea/64","fe80::546b:31ff:fe10:212e/64","fe80::c58:55ff:fe9b:8349/64","fe80::7c84:bff:fe50:1aa7/64","fe80::ecef:7dff:fea9:e6fd/64","fe80::d431:ccff:fe06:695/64","fe80::f835:1bff:fec6:3fb5/64"],"kernel_version":"4.13.0-1016-azure","mac":["00:0d:3a:0c:a1:b8","7e:0a:36:c2:4a:50","66:ba:85:9d:39:cb","9a:f6:f7:ef:a2:11","0e:cd:d9:2c:af:96","70:b3:d5:80:00:05","02:42:22:a5:14:4a","3e:a3:5d:9d:1c:d8","8a:a5:b0:72:09:ca","86:c6:98:c2:ff:fd","6a:1c:8f:71:95:13","5e:09:c0:a6:3a:ea","56:6b:31:10:21:2e","0e:58:55:9b:83:49","7e:84:0b:50:1a:a7","ee:ef:7d:a9:e6:fd","d6:31:cc:06:06:95","fa:35:1b:c6:3f:b5"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.5 LTS (Xenial Xerus)","major":16,"minor":4,"patch":5,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-02-04T14:28:46.942Z INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/etc/metricbeat-6.5.4-linux-x86_64", "exe": "/etc/metricbeat-6.5.4-linux-x86_64/metricbeat", "name": "metricbeat", "pid": 8, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-02-04T14:28:46.070Z"}}}
2019-02-04T14:28:46.942Z INFO instance/beat.go:278 Setup Beat: metricbeat; Version: 6.5.4
2019-02-04T14:28:46.953Z INFO add_cloud_metadata/add_cloud_metadata.go:323 add_cloud_metadata: hosting provider type detected as az, metadata={"instance_id":"ad926b86-07f9-4cfd-95d7-0660c61da114","instance_name":"dcos-agentpool0-34294767--vmss_6","machine_type":"Standard_D4_v3","provider":"az","region":"ukwest"}
2019-02-04T14:28:46.953Z INFO [publisher] pipeline/module.go:110 Beat name: dcos-agentpool0-34294767-000006
2019-02-04T14:28:46.954Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-02-04T14:28:46.954Z INFO kibana/client.go:118 Kibana url: http://10.128.0.6:10112
2019-02-04T14:28:46.957Z INFO [monitoring] log/log.go:152 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":40,"time":{"ms":49}},"total":{"ticks":50,"time":{"ms":68},"value":50},"user":{"ticks":10,"time":{"ms":19}}},"handles":{"limit":{"hard":16384,"soft":16384},"open":5},"info":{"ephemeral_id":"cc741958-9a1e-46c1-a069-ea2cf052a57f","uptime":{"ms":51}},"memstats":{"gc_next":5487296,"memory_alloc":3148736,"memory_total":10341712,"rss":38686720}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":0.14,"15":0.21,"5":0.18,"norm":{"1":0.035,"15":0.0525,"5":0.045}}}}}}
2019-02-04T14:28:46.957Z INFO [monitoring] log/log.go:153 Uptime: 52.497005ms
2019-02-04T14:28:46.957Z INFO [monitoring] log/log.go:130 Stopping metrics logging.
2019-02-04T14:28:46.957Z INFO instance/beat.go:391 metricbeat stopped.
2019-02-04T14:28:46.957Z ERROR instance/beat.go:800 Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://10.128.0.6:10112/api/status: dial tcp 10.128.0.6:10112: connect: connection refused. Response: .
Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://10.128.0.6:10112/api/status: dial tcp 10.128.0.6:10112: connect: connection refused. Response: .

(Mario Castro) #6

I'm not sure about your problem but it really seems like a network config in DCOS.

Let's start from the beginning, try to figure out the address and port of Kibana by trying to access it from a server inside DCOS cluster and using the command line with curl -XGET 10.128.0.6:5601/api/status.

If you get a connection refused error you are simply not pointing to the right address:port and you must first ensure of where it is. If address:port is correctly set, you'll get either a Kibana server is not ready yet or a long JSON response with data.

(mark) #7

Hi,

I have resolved the issue we were missing config within the kibana.yml. the server.name was not set.

thanks for your time.

(Mario Castro) #8

I'm glad that you finally fixed it :slightly_smiling_face:

#9

Hi! If you don't mind me asking, what did you set the name to? Mine says "kibana" by default.

(system) closed #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.