Installing Metricbeat Dashboards on Kibana for Amazon Elasticsearch

mdanner · September 28, 2017, 9:04pm

I'm trying to install the Sample Kibana Dashboards on an instance of the Amazon Elasticsearch Service.

That service is protected by AWS Signature Version 4 Signing.

I issued these commands from an EC2 instance of Ubuntu that has full access to the Amazon Elasticsearch domain via an IAM role.

curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.6.1-amd64.deb
sudo dpkg -i metricbeat-5.6.1-amd64.deb
cd /usr/share/metricbeat
./scripts/import_dashboards -es https://search-xxxxxxxxxx-yyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com/

The last command produces this error:

fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts

Could this be the problem? What's the work around??

warkolm · September 28, 2017, 9:18pm

Unfortunately Amazon provide a customised version of Elasticsearch that removes and limits some of the functionality and while we try to provide compatibility, sometimes it is difficult as they obfuscate these changes.

That said, can you provide the entire command and output from above, including the shell prompts and errors?

mdanner · September 28, 2017, 9:28pm

Hi Mark,

Thanks for having a look at this!

Using username "ubuntu".
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-1035-aws x86_64)

ubuntu@ip-10-0-1-118:~$ cd /usr/share/metricbeat
ubuntu@ip-10-0-1-118:/usr/share/metricbeat$ ./scripts/import_dashboards -es https://search-xxxxxxxxxxxx-yyyyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com/

fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts
Exiting

(Elasticsearch URL obfuscated for security reasons)

mdanner · September 28, 2017, 11:28pm

More info - I relaxed the IAM access policy to allow full access from anywhere:

{   "Version": "2012-10-17",   "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:ap-southeast-2:99999999999:domain/xxxxxxxx/*"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "es:*",
      "Resource": "arn:aws:es:ap-southeast-2:99999999999:domain/xxxxxxxx/*"
    }   
   ] 
}

I also granted the IAM role for the EC2 instance this policy: AmazonESFullAccess

Tried to install the dashboards again - same error!

tudor · September 29, 2017, 5:02am

In Metricbeat 5.6 you can also load the dashboards using metricbeat itself. Can you try like this:

./metricbeat -e -d "*" -setup

And post the output here? I'm hoping the extra debugging will gives us more information about what's going on.

I'm not sure how AWS security works for ES, but importing the dashboards needs write access to the .kibana index.

mdanner · September 29, 2017, 9:37pm

Hi Tudor,

Thanks for the suggestion. Here's the result. Any ideas?

ubuntu@ip-10-0-1-118:/usr/share/metricbeat/bin$ sudo ./metricbeat -e -d "*" -setup -c test.yml -path.config /etc/metricbeat
2017/09/29 21:34:43.977631 beat.go:297: INFO Home path: [/usr/share/metricbeat/bin] Config path: [/etc/metricbeat] Data path: [/usr/share/metricbeat/bin/data] Logs path: [/usr/share/metricbeat/bin/logs]
2017/09/29 21:34:43.977660 beat.go:192: INFO Setup Beat: metricbeat; Version: 5.6.1
2017/09/29 21:34:43.977728 processor.go:44: DBG  Processors:
2017/09/29 21:34:43.977774 metrics.go:23: INFO Metrics logging every 30s
2017/09/29 21:34:43.977789 beat.go:198: DBG  Initializing output plugins
2017/09/29 21:34:43.977975 output.go:258: INFO Loading template enabled. Reading template file: /etc/metricbeat/metricbeat.template.json
2017/09/29 21:34:43.981253 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/metricbeat/metricbeat.template-es2x.json
2017/09/29 21:34:43.984282 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/metricbeat/metricbeat.template-es6x.json
2017/09/29 21:34:43.989571 client.go:128: INFO Elasticsearch url: http://search-xxxxxxxxxx-yyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com:80
2017/09/29 21:34:43.989606 outputs.go:108: INFO Activated elasticsearch as output plugin.
2017/09/29 21:34:43.989621 publish.go:243: DBG  Create output worker
2017/09/29 21:34:43.989707 publish.go:285: DBG  No output is defined to store the topology. The server fields might not be filled.
2017/09/29 21:34:43.989738 publish.go:300: INFO Publisher name: ip-10-0-1-118
2017/09/29 21:34:43.989844 async.go:63: INFO Flush Interval set to: 1s
2017/09/29 21:34:43.989858 async.go:64: INFO Max Bulk Size set to: 50
2017/09/29 21:34:43.989865 async.go:72: DBG  create bulk processing worker (interval=1s, bulk size=50)
2017/09/29 21:34:43.989967 metricbeat.go:28: INFO Register [ModuleFactory:[docker, mongodb, mysql, postgresql, system], MetricSetFactory:[apache/status, ceph/cluster_disk, ceph/cluster_health, ceph/monitor_health, ceph/pool_disk, couchbase/bucket, couchbase/cluster, couchbase/node, docker/container, docker/cpu, docker/diskio, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, haproxy/info, haproxy/stat, jolokia/jmx, kafka/consumergroup, kafka/partition, mongodb/dbstats, mongodb/status, mysql/status, nginx/stubstatus, php_fpm/pool, postgresql/activity, postgresql/bgwriter, postgresql/database, prometheus/collector, prometheus/stats, redis/info, redis/keyspace, system/core, system/cpu, system/diskio, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/process, system/socket, zookeeper/mntr]]
2017/09/29 21:34:43.990190 process.go:68: DBG  process cgroup data collection is enabled, using hostfs=''
2017/09/29 21:34:43.990502 builders.go:56: DBG  mb.NewModules() is returning map[{name:"system", config:{Module:"system", MetricSets:[cpu load filesystem fsstat memory network process], Enabled:true, Hosts:[0 hosts], Period:"10s", Timeout:"10s", Raw:false, Fields:null, FieldsUnderRoot:false, Tags:[]}}:[{name:"cpu", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"load", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"filesystem", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"fsstat", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"memory", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"network", module:"system", hostData:{SanitizedURI:"", Host:""}} {name:"process", module:"system", hostData:{SanitizedURI:"", Host:""}}]]
2017/09/29 21:34:43.990531 wrapper.go:80: DBG  Initializing Module type 'system': *system.Module={name:"system", config:{Module:"system", MetricSets:[cpu load filesystem fsstat memory network process], Enabled:true, Hosts:[0 hosts], Period:"10s", Timeout:"10s", Raw:false, Fields:null, FieldsUnderRoot:false, Tags:[]}}
2017/09/29 21:34:43.990549 processor.go:44: DBG  Processors:
2017/09/29 21:34:43.990591 wrapper.go:96: DBG  Initializing MetricSet type 'system/cpu' for host '': *cpu.MetricSet={name:"cpu", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990616 wrapper.go:96: DBG  Initializing MetricSet type 'system/load' for host '': *load.MetricSet={name:"load", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990643 wrapper.go:96: DBG  Initializing MetricSet type 'system/filesystem' for host '': *filesystem.MetricSet={name:"filesystem", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990667 wrapper.go:96: DBG  Initializing MetricSet type 'system/fsstat' for host '': *fsstat.MetricSet={name:"fsstat", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990691 wrapper.go:96: DBG  Initializing MetricSet type 'system/memory' for host '': *memory.MetricSet={name:"memory", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990716 wrapper.go:96: DBG  Initializing MetricSet type 'system/network' for host '': *network.MetricSet={name:"network", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990741 wrapper.go:96: DBG  Initializing MetricSet type 'system/process' for host '': *process.MetricSet={name:"process", module:"system", hostData:{SanitizedURI:"", Host:""}}
2017/09/29 21:34:43.990907 client.go:128: INFO Elasticsearch url: http://search-xxxxxxxxxx-yyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com:80
2017/09/29 21:34:43.990935 client.go:642: DBG  ES Ping(url=http://search-xxxxxxxxxx-yyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com:80, timeout=1m30s)
2017/09/29 21:34:44.101490 client.go:647: DBG  Ping request failed with: 403 Forbidden
2017/09/29 21:34:44.101511 output.go:88: ERR Error connecting to Elasticsearch: http://search-xxxxxxxxxx-yyyyyyyyyyyyy.ap-southeast-2.es.amazonaws.com:80
2017/09/29 21:34:44.101542 beat.go:346: CRIT Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts
Exiting: Error importing Kibana dashboards: fail to create the Elasticsearch loader: Error creating Elasticsearch client: Couldn't connect to any of the configured Elasticsearch hosts

warkolm · September 29, 2017, 10:00pm

That would be suggestive of a restriction with IAM?

mdanner · October 2, 2017, 10:40pm

I'm pleased to report that this problem is solved. Turns out I was trying to install the wrong version of the dashboards. Amazon Elasticsearch was running version 5.3, but I was trying to install the dashboards for 5.6.

The 5.3 version of the Metricbeat dashboards installed with no errors.

https://www.elastic.co/guide/en/beats/metricbeat/5.3/metricbeat-installation.html

mdanner · October 8, 2017, 5:06pm

I should note that several of the visualizations needed to be tweaked because because they're configured for slightly different field names:

beat.name -> beat.name.keyword
system.process.name -> system.process.name.keyword
system.filesystem.mount_point -> system.filesystem.mount_point.keyword

Many other string fields have the ".keyword" suffix, which the example dashboards don't expect. Edit the dashboard, then edit the affected visualization to fix the problem.

warkolm · October 9, 2017, 12:25am

Did you load the templates as well as the dashboards?

mdanner · October 10, 2017, 6:13am

Good point, Mark. No I didn't. However, loading the templates requires the use of curl, and I'm having difficulty running curl against the Amazon Elasticsearch instance. Apparently curl does not natively support AWS Signature Version 4 Signing.

Any good workarounds for that one?

warkolm · October 10, 2017, 7:08am

https://www.elastic.co/cloud/as-a-service would be my recommendation

Otherwise no, I haven't run into that to be honest with you

mdanner · October 10, 2017, 7:34am

Yep, using Elastic Cloud would definitely be the easiest alternative!

system · November 7, 2017, 7:34am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.