# Kubernetes metadata

**URL:** <https://discuss.elastic.co/t/kubernetes-metadata/90865>\
**Category:** Beats\
**Created:** [June 26, 2017, 7:21pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865 "2017-06-26T19:21:14Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 26, 2017, 7:21pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/1 "2017-06-26T19:21:15Z")

</div>

I'm trying to test the new kubernetes metadata features that were added to beats and therefore would like to pull the docker image for the alpha, but it appears it is not being published.

Could it be published please?

Also appears the github repo links to the logstash documentation - [https://github.com/elastic/beats-docker](https://github.com/elastic/beats-docker)

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 7:36pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/2 "2017-06-26T19:36:50Z")

</div>

Hi @djschny,

Images are published under official elastic docker registry, it should be: `docker.elastic.co/beats/filebeat:6.0.0-alpha2`

Take a look to [https://www.elastic.co/guide/en/beats/filebeat/master/running-on-docker.html](https://www.elastic.co/guide/en/beats/filebeat/master/running-on-docker.html) for complete documentation.

Replace `filebeat` with your desired beat 😉

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 7:39pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/3 "2017-06-26T19:39:06Z")

</div>

Also, as you saw, these are pretty recent additions, so any feedback is welcomed!

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 26, 2017, 8:01pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/4 "2017-06-26T20:01:44Z")

</div>

Thanks, it pulled the image, but the kubernetes module appears to not work:

Exiting: error initializing processors: the processor add\_kubernetes\_metadata doesn't exist

I updated the configuration file as outlined [https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html](https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html)

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 8:12pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/5 "2017-06-26T20:12:56Z")

</div>

I found the issue,

`add_kubernetes_metadata` was renamed recently (will be the definitive name in next version). In alpha2 it was `kubernetes`. The rest of the documentation should be valid. I'm checking our published docs.

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 26, 2017, 8:39pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/6 "2017-06-26T20:39:05Z")

</div>

Thanks! That did the trick, but now receiving the following:

```
2017/06/26 20:32:42.265027 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/06/26 20:32:42.270519 podwatcher.go:87: ERR kubernetes: Watching API eror decode error status: payload is not a kubernetes protobuf object
2017/06/26 20:32:43.271056 podwatcher.go:82: INFO kubernetes: Watching API for pod events
2017/06/26 20:32:43.275282 podwatcher.go:87: ERR kubernetes: Watching API eror decode error status: payload is not a kubernetes protobuf object
2017/06/26 20:32:44.275967 podwatcher.go:82: INFO kubernetes: Watching API for pod events

```

This is with Kubernetes 1.6.6

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 10:05pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/7 "2017-06-26T22:05:51Z")

</div>

Hi,

1.6.X should be supported, could you please share some more details on your setup? I'm interested in how you launch filebeat and what settings are you using. It should be able to access k8s api server, normally this is easy within a pod, but it may require some more params if you are launching it from outside the cluster

Perhaps it's an authentication error with k8s API (with a poor error message)

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 26, 2017, 10:22pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/8 "2017-06-26T22:22:39Z")

</div>

Sure, it's running as a pod. See below for the kubernetes YAML:

```
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat-k8s-logging
  namespace: kube-instrumentation
  labels:
    app: filebeat
    function: logging
spec:
  template:
    metadata:
      labels:
        app: filebeat
        function: logging
      name: filebeat
    spec:
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.0.0-alpha2
        resources:
          limits:
            cpu: 50m
            memory: 50Mi
        securityContext:
          privileged: true
          runAsUser: 0
        env:
          - name: ELASTICSEARCH_URL
            value: http://es-k8s-logging:9200
      terminationGracePeriodSeconds: 30

```

I made sure to set "in\_cluster: true" as well.

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 10:34pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/9 "2017-06-26T22:34:58Z")

</div>

Your pod settings are looking good, I can tell you are using `kube-instrumentation` namespace, you will need to tell filebeat like this:

```auto
kubernetes:
  in_cluster: true
  namespace: kube-instrumentation

```

Apart from that everything is looking good, so I would suggest to enable debug info for kubernetes processor, when launching filebeat add this flag: `-d kubernetes`

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 26, 2017, 10:58pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/10 "2017-06-26T22:58:11Z")

</div>

Thanks for the tip. I tried setting that, but still no luck. With debugging enabled, not much else exciting comes out that would help us. The following is the only items that stood out to me:

```
2017/06/26 22:49:06.519807 podwatcher.go:82: INFO kubernetes: Watching API for pod events
2017/06/26 22:49:06.521560 podwatcher.go:87: ERR kubernetes: Watching API eror decode error status: payload is not a kubernetes protobuf object
2017/06/26 22:49:06.819216 indexing.go:53: DBG Incoming source value: %!(EXTRA string=/var/log/containers/kibana-k8s-logging-2678538301-9h4qj_kube-instrumentation_kibana-79c3aa71931070601554bec365acfc364c2138756b045e63b1e9d5019d82dd53.log)
2017/06/26 22:49:06.819267 indexing.go:59: DBG Using container id: %!(EXTRA string=)
2017/06/26 22:49:07.522742 podwatcher.go:82: INFO kubernetes: Watching API for pod events
2017/06/26 22:49:07.525240 podwatcher.go:87: ERR kubernetes: Watching API eror decode error status: payload is not a kubernetes protobuf object

```

Thanks for the help on this.

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 26, 2017, 11:31pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/11 "2017-06-26T23:31:39Z")

</div>

It seems you are hitting this issue in the client library we use: [https://github.com/ericchiang/k8s/issues/46](https://github.com/ericchiang/k8s/issues/46), do you see that error every 1 second or is it sporadic? I just gave it a try in GKE and it's working for me, but definitely I would like to troubleshoot this issue. Can you share some info about your k8s cluster?

As a side note:

The processor expects you to use /var/lib/docker/containers path, as it's easier to extract containers ids from there

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 27, 2017, 3:27am UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/12 "2017-06-27T03:27:13Z")

</div>

> do you see that error every 1 second or is it sporadic?

It is regular about every 1 second

> Can you share some info about your k8s cluster?

Sure, I'm using [GitHub - kubernetes-retired/kubeadm-dind-cluster: [EOL] A Kubernetes multi-node test cluster based on kubeadm](https://github.com/Mirantis/kubeadm-dind-cluster) to run the cluster locally. All default settings and then the YAML i specified. While diagnosing the problem, I exec into bash on the pod and then run filebeat manually specifying the config until I will get it working and then will abstract that into the YAML.

> The processor expects you to use /var/lib/docker/containers path, as it's easier to extract containers ids from there

Unfortunately that path is empty for me.

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 27, 2017, 10:49am UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/13 "2017-06-27T10:49:54Z")

</div>

I've been checking `kubeadm-dind-cluster` and it seems it's too custom, very oriented to development or light testing.

I'll try to debug the issue as it could affect other scenarios but I would suggest to use a real cluster, or minikube at least, to test this.

Will post here on any finding about this particular issue, thanks for reporting it

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 27, 2017, 4:32pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/14 "2017-06-27T16:32:45Z")

</div>

I deployed into a cluster running on AWS and both good news and bad news:

- Good News - the error messages about the API are gone
- Bad News - no `kubernetes.*` fields are showing up in the documents created in Elasticsearch

I turned on `-d "kubernetes"` and the only new logs that show up are ones like the following:

```auto
2017/06/27 16:32:17.342180 indexing.go:53: DBG Incoming source value: %!(EXTRA string=/var/log/containers/weave-net-vb38v_kube-system_weave-npc-63b03c39a5f7c41b6c76b3c1307ff875ca51724d379b6d3decbb0d31fe4abd30.log)
2017/06/27 16:32:17.347153 indexing.go:59: DBG Using container id: %!(EXTRA string=)
2017/06/27 16:32:18.145902 indexing.go:53: DBG Incoming source value: %!(EXTRA string=/var/log/containers/weave-net-vb38v_kube-system_weave-npc-63b03c39a5f7c41b6c76b3c1307ff875ca51724d379b6d3decbb0d31fe4abd30.log)
2017/06/27 16:32:18.242385 indexing.go:59: DBG Using container id: %!(EXTRA string=)
2017/06/27 16:32:19.143055 indexing.go:53: DBG Incoming source value: %!(EXTRA string=/var/log/containers/weave-net-vb38v_kube-system_weave-npc-63b03c39a5f7c41b6c76b3c1307ff875ca51724d379b6d3decbb0d31fe4abd30.log)
2017/06/27 16:32:19.147099 indexing.go:59: DBG Using container id: %!(EXTRA string=)

```

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 27, 2017, 5:25pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/15 "2017-06-27T17:25:40Z")

</div>

Try with '/var/lib/docker/containers/_/_.log' there

---

<div class="post-metadata">

**Author:** ![djschny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djschny/32/19533_2.png) [@djschny](https://discuss.elastic.co/u/djschny)\
**Post date:** [June 27, 2017, 6:35pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/16 "2017-06-27T18:35:37Z")

</div>

Changing to that path does the trick!

However I'm failing to understand why that should matter. It is optimal for the path to be agnostic of the container engine being used in Kubernetes or where it is located.

> <https://github.com/elastic/beats/blob/fab03dcada0923b9d7b61ac8a33729671391ffa6/filebeat/processor/add_kubernetes_metadata/indexing.go#L29>

Is it assuming that exact path must be present?

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [June 28, 2017, 9:39am UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/17 "2017-06-28T09:39:03Z")

</div>

Certainly this is something we could add, the code expects `/var/lib/docker/containers/` as for now, but to be honest `/var/log/containers/*` are just symlinks to that.

As you can see we have been working on extending our docker and kubernetes support, both for logging and metrics, you can expect more features like this in the future.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2017, 7:21pm UTC](https://discuss.elastic.co/t/kubernetes-metadata/90865/18 "2017-07-17T19:21:19Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.