Filebeat docker input type labels are not shown

[sahinguler]

Hi all,

I use filebeat and logstash for store my all datas. My docker setup and log files are in /home/docker/containers folder. (not in /var/lib/docker path)

The problem is when i was colleting my logs, the docker fields are not shown. (Such as; container.labels.com_docker_swarm_service_name,container.labels.com_docker_swarm_task_name,container.labels.org_label-schema_name,container.labels.org_label-schema_schema-version, container.name,container.image.name etc ).

We know that docker input type was deprecated. Moreover, you suggested to use container input type, unfortunately this type of input hasn't got these labels and our problem is not solved. In our case we used these labels on logstash for separating our datas.

How can we solve this problem ?

How can I see these fieds ?

My filebeat version 7.1.1 and logstash version 7.3.13.

Also my filebeat.yml and logstash.conf files as following.

filebeat.yml

 filebeat.autodiscover:
    providers:
      - type: docker
        templates:
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"

filebeat.inputs:
  #------------------------------ Docker input --------------------------------
  - type: docker
    enabled: true
    encoding: "utf-8"
    containers:
      path: "/home/docker/containers"
      ids:
        - "*"
    multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
    multiline.negate: true
    multiline.match: after
    exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines
    processors:
      - add_host_metadata: ~
      - add_docker_metadata: ~

#=========================== Filebeat inputs ==============================

output.logstash:
  hosts: ["logstash:5044"]

logstash.conf

input {
        beats {
                port => 5044
                ssl  => false
                codec => plain {
            charset => "UTF-8"
                }
        }
}

filter {
    grok { match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:logLevel} %{DATA:logger} %{DATA:threadName} %{DATA:number} %{DATA:module} %{DATA:username}( #indexName%{DATA:type}#)?( #logJson%{DATA:logJson}#)?( #testId%{DATA:testId}#)?( #type%{DATA:type}#)?( #altType%{DATA:altType}#)?( #id%{DATA:id}#)?( #username%{DATA:username}#)? %{DATA:message}"}}
}

output {
    if [fields][log_type] == "deneme"{
        if [log_type] {
            elasticsearch {
                hosts => "elasticsearch:9200"
                manage_template => false
                index => "%{log_type}"
            }
        } else {
            elasticsearch {
                hosts => "elasticsearch:9200"
                manage_template => false
                index => "deneme-logs"
            }
        }
    } else {
        if [container][labels][com_docker_swarm_service_name]=~"test" or [container][labels][com_docker_compose_service]=~"test"{
            if [container][labels][com_docker_swarm_service_name]=~"elasticsearch" or [container][labels][com_docker_swarm_service_name]=~"kibana" or [container][labels][com_docker_swarm_service_name]=~"logstash" or
            [container][labels][com_docker_compose_service]=~"elasticsearch" or [container][labels][com_docker_compose_service]=~"kibana" or [container][labels][com_docker_compose_service]=~"logstash"
            {
                elasticsearch {
                    hosts => "elasticsearch:9200"
                    manage_template => false
                    index => "elk-logs"
                }
            } else {
                if [log_type] {
                    elasticsearch {
                        hosts => "elasticsearch:9200"
                        manage_template => false
                        index => "%{log_type}"
                    }
                } else if [container][labels][com_docker_swarm_service_name]=~"_test" or [container][labels][com_docker_compose_service]=~"_test" {
                    if [container][labels][com_docker_swarm_service_name]=~"_test-gui" or [container][labels][com_docker_compose_service]=~"_test-gui"{
                        elasticsearch {
                            hosts => "elasticsearch:9200"
                            manage_template => false
                            index => "gui"
                        }
                    } else {
                        elasticsearch {
                            hosts => "elasticsearch:9200"
                            manage_template => false
                            index => "backend"
                        }
                    }
                } else {
                    elasticsearch {
                        hosts => "elasticsearch:9200"
                        manage_template => false
                        index => "container"
                    }
                }
            }
        } else {
            elasticsearch {
                hosts => "elasticsearch:9200"
                manage_template => false
                index => "other"
            }
        }
    }
}