Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template

Having trouble getting metricbeats 6.6 running on a Kubernetes cluster.

Used https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml as my deploy file for kubectl. Edited the file for my config then ran

[andy@myserver kubernetes]$ kubectl create -f metricbeat-kubernetes.yaml
configmap/metricbeat-daemonset-config created
configmap/metricbeat-daemonset-modules created
daemonset.extensions/metricbeat created
configmap/metricbeat-deployment-config created
configmap/metricbeat-deployment-modules created
deployment.apps/metricbeat created
clusterrolebinding.rbac.authorization.k8s.io/metricbeat created
clusterrole.rbac.authorization.k8s.io/metricbeat created
serviceaccount/metricbeat created

It wasn't clear in the instructions, but did figure out I still needed to run the docker command to do the install in elasticsearch and kibana, ran:

docker run \
docker.elastic.co/beats/metricbeat:6.6.0 \
setup -E setup.kibana.host=kibana:5601 \
-E output.elasticsearch.hosts=["my-es1:9200"]

Which outputted:

Loaded index template
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

I see the index filter and dashboards loaded into Kibana (but no data flowing in), and if I hit http://my-es1:9200/_template?pretty I see the index template (below is just a snippet, it's very long):

"metricbeat-6.6.0" : {
    "order" : 1,
    "index_patterns" : [
      "metricbeat-6.6.0-*"
    ],
    "settings" : {
      "index" : {
        "number_of_routing_shards" : "30",
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "doc" : {
        "_meta" : {
          "version" : "6.6.0"
        },
        "date_detection" : false,
        "dynamic_templates" : [
          {
            "fields" : {
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string",
              "path_match" : "fields.*"
            }
          },
          {
            "docker.container.labels" : {
              "mapping" : {
                "type" : "keyword"
              },
              "match_mapping_type" : "string",
              "path_match" : "docker.container.labels.*"
            }
          },
          {
            "system.process.env" : {
              "match_mapping_type" : "string",
              "path_match" : "system.process.env.*",
              "mapping" : {
                "type" : "keyword"
              }
            }
          },
          {
            "system.process.cgroup.cpuacct.percpu" : {

However I'm getting errors from the metricbeat pods trying to hit the elasticsearch server:

019-02-06T02:51:25.009Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://my-es1:9200)): Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [doc]: No handler for type [alias] declared on field [hostname]","caused_by":{"type":"mapper_parsing_exception","reason":"No handler for type [alias] declared on field [hostname]"}},

The above error is huge, and is always followed by:

2019-02-06T02:51:19.825Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://my-es1:9200)) with 12 reconnect attempt(s)
2019-02-06T02:51:19.825Z        INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-02-06T02:51:19.825Z        INFO    [publish]       pipeline/retry.go:191     done
2019-02-06T02:51:19.825Z        INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
2019-02-06T02:51:19.825Z        INFO    [publish]       pipeline/retry.go:168     done
2019-02-06T02:51:19.826Z        INFO    elasticsearch/client.go:752     Connected to Elasticsearch version 6.3.2
2019-02-06T02:51:19.827Z        INFO    template/load.go:83     Loading template for Elasticsearch version: 6.3.2
2019-02-06T02:51:40.158Z        INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":3530,"time":{"ms":222}},"total":{"ticks":9460,"time":{"ms":630},"value":9460},"user":{"ticks":5930,"time":{"ms":408}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":9},"info":{"ephemeral_id":"13a1ae58-7537-4b46-a526-c7a49426ce93","uptime":{"ms":450079}},"memstats":{"gc_next":34733184,"memory_alloc":25054136,"memory_total":1200744072,"rss":6520832}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":807},"write":{"bytes":97911}},"pipeline":{"clients":6,"events":{"active":1357,"filtered":1,"published":90,"retry":26,"total":91}}},"metricbeat":{"kubernetes":{"container":{"events":3,"failures":3},"node":{"events":3,"failures":3},"pod":{"events":3,"failures":3},"system":{"events":3,"failures":3},"volume":{"events":3,"failures":3}},"system":{"cpu":{"events":3,"success":3},"filesystem":{"events":1,"success":1},"fsstat":{"events":1,"success":1},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":42,"success":42},"process":{"events":20,"success":20},"process_summary":{"events":3,"success":3}}},"system":{"load":{"1":0.43,"15":0.57,"5":0.66,"norm":{"1":0.215,"15":0.285,"5":0.33}}}}}}

I've tried deleting the index template and recreating it, but nothing changes. Any ideas?

Hi @andymelichar

It seems like a version mismatch error. The error you're getting is unrelated with kubernetes. Can you also share wich version of Elasticsearch and Kibana you are using?

By the way, can you also paste your metricbeat configuration and which modules you have enabled / want to have enabled.

@Mario_Castro I had wondered if might be a version issue. I pulled down the metricbeat 6.5 yaml file and ran it, everything worked perfectly.

FWIW:

{
  "name" : "my-es1",
  "cluster_name" : "my-logger",
  "cluster_uuid" : "xxxxxxxxxxxx",
  "version" : {
    "number" : "6.3.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "053779d",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

Kibana is also 6.3.2

Looks like this is a known issue: https://github.com/elastic/beats/issues/9989

I'm glad that you fixed it. Thanks for sharing!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.