Filebeat to elasticsearch in Kubernetes

[clule404]

Hi everyone,

I have some problems with Filebeat -> Elasticsearch -> Kibana in Kubernetes.
I get my logs to show up in Kibana, that's not the problem.
The problem is, that every Log that shows up just has the following information:

beat.hostname
beat.name
beat.version 	
kubernetes.container.name 	
kubernetes.namespace 	
kubernetes.node.name 	
kubernetes.pod.name 	
message

Now I want to use the Filebeat Dashboard for NGINX logs but it needs fields like nginx.access.geoip.location and so on. But I guess that is only possible if the nginx is not running in Kubernetes.
Is that right or do I just need to change some configuration in filebeat?

These two ConfigMaps I am using in Kuberentes:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.config:
      prospectors:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    processors:
      - add_cloud_metadata:

              #cloud.id: ${ELASTIC_CLOUD_ID}
              #cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      #bulk_max_size: 1000
      #username: ${ELASTICSEARCH_USERNAME}
      #password: ${ELASTICSEARCH_PASSWORD}
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---

Thanks for any help

[exekias (Carlos Pérez Aradros)]

Hi @clule404,

It's actually possible, have a look at autodiscover features, there is an example for redis: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_docker_2

You would have to add something like this to filebeat.yml (I haven't tested it):

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            equals:
              docker.container.image: nginx
          config:
            - module: nginx
              access:
                prospector:
                  type: docker
                  containers.ids:
                  - "${data.docker.container.id}"