Having trouble getting metricbeats 6.6 running on a Kubernetes cluster.
Used https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml as my deploy file for kubectl. Edited the file for my config then ran
[andy@myserver kubernetes]$ kubectl create -f metricbeat-kubernetes.yaml
configmap/metricbeat-daemonset-config created
configmap/metricbeat-daemonset-modules created
daemonset.extensions/metricbeat created
configmap/metricbeat-deployment-config created
configmap/metricbeat-deployment-modules created
deployment.apps/metricbeat created
clusterrolebinding.rbac.authorization.k8s.io/metricbeat created
clusterrole.rbac.authorization.k8s.io/metricbeat created
serviceaccount/metricbeat created
It wasn't clear in the instructions, but did figure out I still needed to run the docker command to do the install in elasticsearch and kibana, ran:
docker run \
docker.elastic.co/beats/metricbeat:6.6.0 \
setup -E setup.kibana.host=kibana:5601 \
-E output.elasticsearch.hosts=["my-es1:9200"]
Which outputted:
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
I see the index filter and dashboards loaded into Kibana (but no data flowing in), and if I hit http://my-es1:9200/_template?pretty I see the index template (below is just a snippet, it's very long):
"metricbeat-6.6.0" : {
"order" : 1,
"index_patterns" : [
"metricbeat-6.6.0-*"
],
"settings" : {
"index" : {
"number_of_routing_shards" : "30",
"mapping" : {
"total_fields" : {
"limit" : "10000"
}
},
"refresh_interval" : "5s"
}
},
"mappings" : {
"doc" : {
"_meta" : {
"version" : "6.6.0"
},
"date_detection" : false,
"dynamic_templates" : [
{
"fields" : {
"mapping" : {
"type" : "keyword"
},
"match_mapping_type" : "string",
"path_match" : "fields.*"
}
},
{
"docker.container.labels" : {
"mapping" : {
"type" : "keyword"
},
"match_mapping_type" : "string",
"path_match" : "docker.container.labels.*"
}
},
{
"system.process.env" : {
"match_mapping_type" : "string",
"path_match" : "system.process.env.*",
"mapping" : {
"type" : "keyword"
}
}
},
{
"system.process.cgroup.cpuacct.percpu" : {
However I'm getting errors from the metricbeat pods trying to hit the elasticsearch server:
019-02-06T02:51:25.009Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://my-es1:9200)): Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [alias] declared on field [hostname]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}},
The above error is huge, and is always followed by:
2019-02-06T02:51:19.825Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://my-es1:9200)) with 12 reconnect attempt(s)
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:191 done
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:168 done
2019-02-06T02:51:19.826Z INFO elasticsearch/client.go:752 Connected to Elasticsearch version 6.3.2
2019-02-06T02:51:19.827Z INFO template/load.go:83 Loading template for Elasticsearch version: 6.3.2
2019-02-06T02:51:40.158Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3530,"time":{"ms":222}},"total":{"ticks":9460,"time":{"ms":630},"value":9460},"user":{"ticks":5930,"time":{"ms":408}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":9},"info":{"ephemeral_id":"13a1ae58-7537-4b46-a526-c7a49426ce93","uptime":{"ms":450079}},"memstats":{"gc_next":34733184,"memory_alloc":25054136,"memory_total":1200744072,"rss":6520832}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":807},"write":{"bytes":97911}},"pipeline":{"clients":6,"events":{"active":1357,"filtered":1,"published":90,"retry":26,"total":91}}},"metricbeat":{"kubernetes":{"container":{"events":3,"failures":3},"node":{"events":3,"failures":3},"pod":{"events":3,"failures":3},"system":{"events":3,"failures":3},"volume":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":42,"success":42},"process":{"events":20,"success":20},"process_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.43,"15":0.57,"5":0.66,"norm":{"1":0.215,"15":0.285,"5":0.33}}}}}}
I've tried deleting the index template and recreating it, but nothing changes. Any ideas?