# Autodiscover hints.enabled logs all pods

**URL:** <https://discuss.elastic.co/t/autodiscover-hints-enabled-logs-all-pods/203669>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 15, 2019, 2:13pm UTC](https://discuss.elastic.co/t/autodiscover-hints-enabled-logs-all-pods/203669 "2019-10-15T14:13:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![symjar](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@symjar](https://discuss.elastic.co/u/symjar)\
**Post date:** [October 15, 2019, 2:13pm UTC](https://discuss.elastic.co/t/autodiscover-hints-enabled-logs-all-pods/203669/1 "2019-10-15T14:13:20Z")

</div>

Hey!

I've attempted to implement kubernetes logging to elasticsearch.

What I want to achieve:

1. Ship all the logs from pods in the default namespace.
2. Configure templates per pod type

[That](https://github.com/elastic/beats/blob/7.3/deploy/kubernetes/filebeat-kubernetes.yaml) was my first attempt and it works (ships all pods ). Then I've decided to enable autodiscover option and limit it to only log pods from default namespace. Here is my config:

```
 filebeat.autodiscover:
 providers:
   - type: kubernetes
     host: ${NODE_NAME}
     #hints.enabled: true
     templates:
      - condition:
          equals:
            kubernetes.namespace: default
        config:
            type: container
            paths:
              - /var/log/container/*-${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_host_metadata:

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
  username: ${ELASTICSEARCH_USERNAME}
  password: ${ELASTICSEARCH_PASSWORD}

```

Unfortunately it doesn't log anything at all. Even though I have a pod in the default namespace there is nothing shipped to ES. If I uncomment `hints.enabled` it logs everything (both logs from my desired pod and filebeat logs - filebeat is form the kube-system namespace)

Is it enough information to determinate what might be a problem?

---

<div class="post-metadata">

**Author:** ![symjar](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@symjar](https://discuss.elastic.co/u/symjar)\
**Post date:** [October 16, 2019, 12:27pm UTC](https://discuss.elastic.co/t/autodiscover-hints-enabled-logs-all-pods/203669/2 "2019-10-16T12:27:53Z")

</div>

What I've been able to find out is that logs in my cluster are located in a different folder

For example:  
`/var/lib/docker/containers/64b4e054eb0007977ec7124a326dbbd3c257c9e75ceb3b8c0d611e7851f10a4d/64b4e054eb0007977ec7124a326dbbd3c257c9e75ceb3b8c0d611e7851f10a4d-json.log`

I've just a newly created AWS EKS cluster (Kubernetes version 1.14), newly started ES and Kibana (version 7.4), running outside of the cluster.

In the cluster I've deployed this:

```auto
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-boot-deployment
  annotations:
    co.elastic.logs/enabled: false
    # co.elastic.logs/module: logstash
spec:
  selector:
    matchLabels:
      app: spring-boot-app
  replicas: 3
  template:
    metadata:
      labels:
        app: spring-boot-app
    spec:
      containers:
      - name: spring-boot-app
        image: myimage
        ports:
        - containerPort: 8080
          name: server
        - containerPort: 8081
          name: management

```

My current configmap for filebeat (v 7.40, running in the cluster:

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    # logging.level: debug
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
---

```

Even though I annotated my deployment with `co.elastic.logs/enabled: false` Filebeat still picks it up and ships to my ES instance.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 13, 2019, 12:27pm UTC](https://discuss.elastic.co/t/autodiscover-hints-enabled-logs-all-pods/203669/3 "2019-11-13T12:27:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.