# Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template
**URL:** https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209
**Category:** Beats
**Tags:** metricbeat
**Created:** [February 6, 2019, 3:14am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209 "2019-02-06T03:14:59Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@andymelichar](https://discuss.elastic.co/u/andymelichar)
#### Post date: [February 6, 2019, 3:14am UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209/1 "2019-02-06T03:14:59Z")
Having trouble getting metricbeats 6.6 running on a Kubernetes cluster.
Used [https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml](https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml) as my deploy file for kubectl. Edited the file for my config then ran
```
[andy@myserver kubernetes]$ kubectl create -f metricbeat-kubernetes.yaml
configmap/metricbeat-daemonset-config created
configmap/metricbeat-daemonset-modules created
daemonset.extensions/metricbeat created
configmap/metricbeat-deployment-config created
configmap/metricbeat-deployment-modules created
deployment.apps/metricbeat created
clusterrolebinding.rbac.authorization.k8s.io/metricbeat created
clusterrole.rbac.authorization.k8s.io/metricbeat created
serviceaccount/metricbeat created
```
It wasn't clear in the instructions, but did figure out I still needed to run the docker command to do the install in elasticsearch and kibana, ran:
```
docker run \
docker.elastic.co/beats/metricbeat:6.6.0 \
setup -E setup.kibana.host=kibana:5601 \
-E output.elasticsearch.hosts=["my-es1:9200"]
```
Which outputted:
```
Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
```
I see the index filter and dashboards loaded into Kibana (but no data flowing in), and if I hit [http://my-es1:9200/\_template?pretty](http://my-es1:9200/_template?pretty) I see the index template (below is just a snippet, it's very long):
```
"metricbeat-6.6.0" : {
"order" : 1,
"index_patterns" : [
"metricbeat-6.6.0-*"
],
"settings" : {
"index" : {
"number_of_routing_shards" : "30",
"mapping" : {
"total_fields" : {
"limit" : "10000"
}
},
"refresh_interval" : "5s"
}
},
"mappings" : {
"doc" : {
"_meta" : {
"version" : "6.6.0"
},
"date_detection" : false,
"dynamic_templates" : [
{
"fields" : {
"mapping" : {
"type" : "keyword"
},
"match_mapping_type" : "string",
"path_match" : "fields.*"
}
},
{
"docker.container.labels" : {
"mapping" : {
"type" : "keyword"
},
"match_mapping_type" : "string",
"path_match" : "docker.container.labels.*"
}
},
{
"system.process.env" : {
"match_mapping_type" : "string",
"path_match" : "system.process.env.*",
"mapping" : {
"type" : "keyword"
}
}
},
{
"system.process.cgroup.cpuacct.percpu" : {
```
However I'm getting errors from the metricbeat pods trying to hit the elasticsearch server:
`019-02-06T02:51:25.009Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://my-es1:9200)): Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [alias] declared on field [hostname]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}},`
The above error is huge, and is always followed by:
```
2019-02-06T02:51:19.825Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://my-es1:9200)) with 12 reconnect attempt(s)
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:191 done
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-02-06T02:51:19.825Z INFO [publish] pipeline/retry.go:168 done
2019-02-06T02:51:19.826Z INFO elasticsearch/client.go:752 Connected to Elasticsearch version 6.3.2
2019-02-06T02:51:19.827Z INFO template/load.go:83 Loading template for Elasticsearch version: 6.3.2
2019-02-06T02:51:40.158Z INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3530,"time":{"ms":222}},"total":{"ticks":9460,"time":{"ms":630},"value":9460},"user":{"ticks":5930,"time":{"ms":408}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":9},"info":{"ephemeral_id":"13a1ae58-7537-4b46-a526-c7a49426ce93","uptime":{"ms":450079}},"memstats":{"gc_next":34733184,"memory_alloc":25054136,"memory_total":1200744072,"rss":6520832}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":807},"write":{"bytes":97911}},"pipeline":{"clients":6,"events":{"active":1357,"filtered":1,"published":90,"retry":26,"total":91}}},"metricbeat":{"kubernetes":{"container":{"events":3,"failures":3},"node":{"events":3,"failures":3},"pod":{"events":3,"failures":3},"system":{"events":3,"failures":3},"volume":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":42,"success":42},"process":{"events":20,"success":20},"process_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.43,"15":0.57,"5":0.66,"norm":{"1":0.215,"15":0.285,"5":0.33}}}}}}
```
I've tried deleting the index template and recreating it, but nothing changes. Any ideas?
---
### Author:  [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)
#### Post date: [February 6, 2019, 12:05pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209/2 "2019-02-06T12:05:01Z")
Hi @andymelichar
It seems like a version mismatch error. The error you're getting is unrelated with kubernetes. Can you also share wich version of Elasticsearch and Kibana you are using?
By the way, can you also paste your metricbeat configuration and which modules you have enabled / want to have enabled.
---
### Author:  [@andymelichar](https://discuss.elastic.co/u/andymelichar)
#### Post date: [February 6, 2019, 1:24pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209/3 "2019-02-06T13:24:35Z")
@Mario_Castro I had wondered if might be a version issue. I pulled down the metricbeat 6.5 yaml file and ran it, everything worked perfectly.
FWIW:
```
{
"name" : "my-es1",
"cluster_name" : "my-logger",
"cluster_uuid" : "xxxxxxxxxxxx",
"version" : {
"number" : "6.3.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "053779d",
"build_date" : "2018-07-20T05:20:23.451332Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
```
Kibana is also 6.3.2
Looks like this is a known issue: [https://github.com/elastic/beats/issues/9989](https://github.com/elastic/beats/issues/9989)
---
### Author:  [@Mario\_Castro](https://discuss.elastic.co/u/Mario_Castro)
#### Post date: [February 6, 2019, 3:46pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209/4 "2019-02-06T15:46:09Z")
I'm glad that you fixed it. Thanks for sharing!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 6, 2019, 3:46pm UTC](https://discuss.elastic.co/t/connection-marked-as-failed-because-the-onconnect-callback-failed-error-loading-elasticsearch-template/167209/5 "2019-03-06T15:46:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.