# Kubernetes add\_kubernetes\_metadata not able to send

**URL:** <https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 15, 2020, 2:28pm UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842 "2020-05-15T14:28:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![vasu\_devan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasu_devan/32/68427_2.png) [@vasu\_devan](https://discuss.elastic.co/u/vasu_devan)\
**Post date:** [May 15, 2020, 2:28pm UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/1 "2020-05-15T14:28:29Z")

</div>

Hi Team,

`I am planning to send custom path logs(example: /var/log/nginx/*logs) with cloud and kubernetes metadata.`

`sample App deployed:`

````auto
kind: Deployment
metadata:
  name: nginx-deployment-test
  namespace: vasu
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80
        volumeMounts:
        - name: shared-logs
          mountPath: /var/log/nginx
      volumes:
      - name: shared-logs
        hostPath:
          path: /var/log/nginx
---
apiVersion: v1
kind: Service
metadata:
  name: my-nginx-service
  namespace: vasu
spec:
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 8081
      targetPort: 80```

`Filebeat complete config :`

```apiVersion: v1
kind: ServiceAccount
metadata:
  name: teststack-shipper
  namespace: vasu
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teststack-shipper
rules:
- apiGroups:
  - ""
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: teststack-shipper
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teststack-shipper
subjects:
- kind: ServiceAccount
  name: teststack-shipper
  namespace: vasu
---
apiVersion: v1
data:
  teststack-config.yml: |
      #=========================== Filebeat prospectors =============================
      fields_under_root: false
      fields:
        product: ${PRODUCT}
        env: ${ENVIRONMENT}
        group: ${GROUP}
        region: ${AWS_REGION}
        cluster: ${K8S_CLUSTER_NAME}
        node_name: ${NODE_NAME} 
      max_procs: 1
      filebeat.inputs:
      - type: log
        tail_files: true
        fields_under_root: false
        enabled: true
        encoding: utf-8
        max_bytes: 65536
        fields: 
          name: nginx
        paths: 
          - /var/log/nginx/*.log
      

      processors:
        - add_cloud_metadata: ~
      #============================= Filebeat modules ===============================

      path.data: /teststack-shipper/data
      path.logs: /teststack-shipper/logs

      filebeat.config.modules:
        # Glob pattern for configuration loading
        path: ${path.config}/modules.d/*.yml

        # Set to true to enable config reloading
        reload.enabled: false

        # Period on which files under path should be checked for changes
        #reload.period: 10s

      #================================ Outputs =====================================

      # Configure what output to use when sending the data collected by the beat.

      output.http:
        hosts: ${TESTSTACK_SHIPPER_HOST}
        compression_level: ${TESTSTACK_SHIPPER_COMPRESSION_LEVEL}
        worker: ${TESTSTACK_SHIPPER_WORKER}
        batch_size: ${TESTSTACK_SHIPPER_BATCH_SIZE}
        auth_token: ${TESTSTACK_SHIPPER_AUTH_TOKEN}
        batch_publish: true

      #output.logstash:
        # hosts: ["10.99.111.60:5044"]
          #worker: 4

      # output.file:
      # path: "/tmp/filebeat"
      # filename: filebeat

      #================================ Logging =====================================

      # Sets log level. The default log level is info.
      # Available log levels are: error, warning, info, debug
      # logging.level: debug

      # At debug level, you can selectively enable logging only for some components.
      # To enable all selectors use ["*"]. Examples of other selectors are "beat",
      # "publish", "service".
      # logging.selectors: ["http"]
kind: ConfigMap
metadata:
  name: teststack-shipper-config-node-b6566ggm6g
  namespace: vasu
---
apiVersion: v1
data:
  AWS_INFRA_SECRETS: stagfpteststack
  AWS_REGION: us-east-1
  ENVIRONMENT: staging
  GROUP: k8s
  TESTSTACK_SHIPPER_BATCH_SIZE: "40"
  TESTSTACK_SHIPPER_COMPRESSION_LEVEL: "0"
  TESTSTACK_SHIPPER_HOST: http://vpce-09ab.aws.com/staging-fp/bulk
  TESTSTACK_SHIPPER_WORKER: "8"
  PRODUCT: staging-fcp
kind: ConfigMap
metadata:
  name: teststack-shipper-env
  namespace: vasu
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    k8s-app: teststack-shipper
  name: teststack-shipper
  namespace: vasu
spec:
  selector:
    matchLabels:
      name: teststack-shipper
  template:
    metadata:
      labels:
        name: teststack-shipper
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: NODE_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        envFrom:
        - configMapRef:
            name: teststack-shipper-env
        image: 2134.dkr.ecr.us-east-1.amazonaws.com/teststack:6.7.2
        imagePullPolicy: Always
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -c
              - sleep 60
        name: teststack-shipper
        resources:
          limits:
            memory: 2000Mi
          requests:
            cpu: 100m
            ephemeral-storage: 10Gi
            memory: 500Mi
        securityContext:
          runAsUser: 0
        volumeMounts:
        - mountPath: /var/log
          name: varlog
          readOnly: true
        - mountPath: /var/lib/docker
          name: varlibdocker
          readOnly: true
        - mountPath: /etc/teststack-shipper
          name: teststack-shipper-config-node
      hostNetwork: false
      hostPID: false
      priorityClassName: high-priority
      serviceAccountName: teststack-shipper
      terminationGracePeriodSeconds: 120
      volumes:
      - hostPath:
          path: /var/log
        name: varlog
      - hostPath:
          path: /var/lib/docker
        name: varlibdocker
      - configMap:
          name: teststack-shipper-config-node-b6566ggm6g
        name: teststack-shipper-config-node
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate ```

`Able to send add_cloud_metadata value , but unable to send add_kubernetes_metadata.`

`Please help on this.`
````

---

<div class="post-metadata">

**Author:** ![vasu\_devan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasu_devan/32/68427_2.png) [@vasu\_devan](https://discuss.elastic.co/u/vasu_devan)\
**Post date:** [May 20, 2020, 5:53am UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/2 "2020-05-20T05:53:44Z")

</div>

Hi Team,

Waiting for your inputs, please help on this.

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [May 20, 2020, 10:38am UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/3 "2020-05-20T10:38:39Z")

</div>

I don't see any definition for the processor. Did you try to add this one? [https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html](https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html)

---

<div class="post-metadata">

**Author:** ![vasu\_devan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasu_devan/32/68427_2.png) [@vasu\_devan](https://discuss.elastic.co/u/vasu_devan)\
**Post date:** [May 21, 2020, 1:20pm UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/4 "2020-05-21T13:20:46Z")

</div>

Hi @mtojek

`Yes I added processor as below. `

```
 processors:
    - add_cloud_metadata: ~
    - add_kubernetes_metadata:
        in_cluster: true

```

`We are receiving the cloud_metadata as below and unable to get kubernetes_metadata.`

````auto
  "provider": "ec2",
  "region": "us-east-1",
  "availability_zone": "us-east-1d",
  "instance_id": "i-0cb35e306b",
  "machine_type": "r5.2xlarge"
}```

`our filebeat version: 6.7.2.`

`pls let me know if anything needs to check.`
````

---

<div class="post-metadata">

**Author:** ![vasu\_devan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vasu_devan/32/68427_2.png) [@vasu\_devan](https://discuss.elastic.co/u/vasu_devan)\
**Post date:** [May 27, 2020, 6:25am UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/5 "2020-05-27T06:25:19Z")

</div>

Hi @mtojek

Waiting for your input.help on this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 24, 2020, 6:25am UTC](https://discuss.elastic.co/t/kubernetes-add-kubernetes-metadata-not-able-to-send/232842/6 "2020-06-24T06:25:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.