Filebeat: docker autodiscover problem

111207 · April 4, 2022, 6:08am

I need to using filebeat auto discover to input the docker container log
My container log name pantsel/konga
This is my config in filebeat.yml

#==============Using Autodiscover=======================
filebeat.autodiscover:
        providers:
                - type: docker
                  templates:
                          - condition:
                                  contains:
                                          docker.container.image: pantsel/konga
                                          config:
                                                  - type: container
                                                    paths:
                                                            - '/var/lib/docker/containers/${data.docker.container.id}/*.log'
                                                    exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

kvch · April 5, 2022, 12:45pm

Could you please share the logs of Filebeat?

Also, the indentation of your configuration looks wild. In some cases Filebeat cannot parse configuration that are not indented correctly. Could you please try with a correct indentation?

#==============Using Autodiscover=======================
 filebeat.autodiscover:
   providers:
     - type: docker
       templates:
         - condition:
           contains:
             docker.container.image: pantsel/konga
       config:
         - type: container
           paths:
             - '/var/lib/docker/containers/${data.docker.container.id}/*.log'
           exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

111207 · April 6, 2022, 5:47pm

I'm tried your configuration for autodiscover

But docker log not show in kibana

This is my filebeat.log
2022-04-06T17:45:03.507Z DEBUG [input] input-cursor/clean.go:60 Start store cleanup {"input_type": "journald"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:69 No entries to remove were found {"input_type": "journald"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:70 Done store cleanup {"input_type": "journald"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:60 Start store cleanup {"input_type": "o365audit"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:69 No entries to remove were found {"input_type": "o365audit"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:70 Done store cleanup {"input_type": "o365audit"} 2022-04-06T17:45:03.508Z DEBUG [input] input-logfile/clean.go:60 Start store cleanup {"input_type": "filestream"} 2022-04-06T17:45:03.508Z DEBUG [input] input-logfile/clean.go:69 No entries to remove were found {"input_type": "filestream"} 2022-04-06T17:45:03.508Z DEBUG [input] input-logfile/clean.go:70 Done store cleanup {"input_type": "filestream"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:60 Start store cleanup {"input_type": "httpjson"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:69 No entries to remove were found {"input_type": "httpjson"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:70 Done store cleanup {"input_type": "httpjson"} 2022-04-06T17:45:03.507Z DEBUG [input] input-cursor/clean.go:60 Start store cleanup {"input_type": "httpjson"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:69 No entries to remove were found {"input_type": "httpjson"} 2022-04-06T17:45:03.508Z DEBUG [input] input-cursor/clean.go:70 Done store cleanup {"input_type": "httpjson"} 2022-04-06T17:45:05.223Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_create: /bin/sh -c pg_isready -U postgres 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_create: /bin/sh -c pg_isready -U postgres {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105222568311} 2022-04-06T17:45:05.223Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_start: /bin/sh -c pg_isready -U postgres 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_start: /bin/sh -c pg_isready -U postgres {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105222671442} 2022-04-06T17:45:05.223Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_create: /bin/sh -c pg_isready -U postgres 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_create: /bin/sh -c pg_isready -U postgres {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105222568311} 2022-04-06T17:45:05.223Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_start: /bin/sh -c pg_isready -U postgres 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_start: /bin/sh -c pg_isready -U postgres {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105222671442} 2022-04-06T17:45:05.393Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_die 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_die {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 exitCode:0 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105392033334} 2022-04-06T17:45:05.393Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_die 615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 postgres:9.6 container exec_die {615c07badd744785442a68d64b65a2c7e59b09a2a71769ebc4fef76456404d63 map[com.docker.compose.config-hash:e55dac92749fdbcffbf53a03a1203751a507f0edc86e7e6591bf61ccaa01fb01 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:keycloak-db com.docker.compose.version:1.25.5 execID:39c8366b7cbb9263633fbd2ee9f099eb31fdaa4033f1c942a609649967bcf852 exitCode:0 image:postgres:9.6 name:keycloak-db]} local 1649267105 1649267105392033334} 2022-04-06T17:45:06.593Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_create: /bin/sh -c pg_isready -U postgres 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_create: /bin/sh -c pg_isready -U postgres {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b image:postgres:9.6 name:kong-database]} local 1649267106 1649267106593273509} 2022-04-06T17:45:06.593Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_start: /bin/sh -c pg_isready -U postgres 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_start: /bin/sh -c pg_isready -U postgres {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b image:postgres:9.6 name:kong-database]} local 1649267106 1649267106593387639} 2022-04-06T17:45:06.594Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_create: /bin/sh -c pg_isready -U postgres 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_create: /bin/sh -c pg_isready -U postgres {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b image:postgres:9.6 name:kong-database]} local 1649267106 1649267106593273509} 2022-04-06T17:45:06.594Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_start: /bin/sh -c pg_isready -U postgres 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_start: /bin/sh -c pg_isready -U postgres {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b image:postgres:9.6 name:kong-database]} local 1649267106 1649267106593387639} 2022-04-06T17:45:06.760Z DEBUG [docker] docker/watcher.go:279 Got a new docker event: {exec_die 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_die {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b exitCode:0 image:postgres:9.6 name:kong-database]} local 1649267106 1649267106759671157} 2022-04-06T17:45:06.760Z DEBUG [add_docker_metadata] docker/watcher.go:279 Got a new docker event: {exec_die 950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 postgres:9.6 container exec_die {950d8d729aaf7814eddd2abe57f80501836b0afb393ab81599d2c9d0da30ba63 map[com.docker.compose.config-hash:1357bb9dd61524791551e26625e03a31d74fcdecfe4d8f151a1c0dbd64413679 com.docker.compose.container-number:1 com.docker.compose.oneoff:False com.docker.compose.project:kong com.docker.compose.project.config_files:docker-compose.yaml com.docker.compose.project.working_dir:/root/kong com.docker.compose.service:kong-database com.docker.compose.version:1.25.5 execID:1445cc51e10b557b0334be618803fd9fe9eb3d5de14943b315ba779032a1036b exitCode:0 image:postgres:9.6 name:kong-database]} local 1649267106 1649267106759671157}

111207 · April 25, 2022, 6:08am

Hi kvch

Please advise to resolve this. Please let me know if you need another information

system · May 23, 2022, 8:08am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.