Filebeat data not indexed

Elastic Stack Beats
,
1

Hi,
i'm using filebeat 7.15.1 running as a docker container with the following configuration:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 60s

filebeat.autodiscover:
  providers:
    - type: docker
      hits.enabled: true
      json.key_under_root: true
      labels.dedot: true
      templates:
        - condition:
            contains:
                  docker.container.labels.filebeatenabled: "true"
          config:
            - type: container
              format: docker
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              processors:
                - add_docker_metadata: ~      
                - decode_json_fields:
                    when.equals:
                      docker.container.labels.decodeLogEventToJsonObject: "true"
                    fields: ["message"]
                    target: ""
                    overwrite_keys: true

output:
  elasticsearch:
    enabled: true
    hosts: ["http://elk:9200"]

setup.kibana.host: 'elk-2:5601'
logging.level: debug 
logging.metrics.enabled: false

i've succefully executed the filebeat setup, but the elastic index is empty.

Filebeat seems to work fine, debug logs shows what i suppose is a correctly discovered container and log data:

filebeat_1             | 2021-12-02T16:37:11.697Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:172	Got a start event.	{"autodiscover.event": {"config":[{},{},{},{}],"container":{"id":"70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f","image":{"name":"egpal-web"},"labels":{"com":{"docker":{"compose":{"config-hash":"82e0fa1a5b41e72b39435cfa02775329be659cb0f0f52f9638b1a3e5b25e72eb","container-number":"1","oneoff":"False","project":{"config_files":"docker-compose.yml","value":"egpal","working_dir":"/home/asegpal/app/egpal"},"service":"egpal-web","version":"1.29.2"}}},"filebeatenabled":"true","maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"name":"egpal_egpal-web_1"},"docker":{"container":{"id":"70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f","image":"egpal-web","labels":{"com":{"docker":{"compose":{"config-hash":"82e0fa1a5b41e72b39435cfa02775329be659cb0f0f52f9638b1a3e5b25e72eb","container-number":"1","oneoff":"False","project":{"config_files":"docker-compose.yml","value":"egpal","working_dir":"/home/asegpal/app/egpal"},"service":"egpal-web","version":"1.29.2"}}},"filebeatenabled":"true","maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"name":"egpal_egpal-web_1"}},"host":"172.21.0.11","id":"70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f","meta":{"container":{"id":"70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f","image":{"name":"egpal-web"},"name":"egpal_egpal-web_1"},"docker":{"container":{"labels":{"com_docker_compose_config-hash":"82e0fa1a5b41e72b39435cfa02775329be659cb0f0f52f9638b1a3e5b25e72eb","com_docker_compose_container-number":"1","com_docker_compose_oneoff":"False","com_docker_compose_project":"egpal","com_docker_compose_project_config_files":"docker-compose.yml","com_docker_compose_project_working_dir":"/home/asegpal/app/egpal","com_docker_compose_service":"egpal-web","com_docker_compose_version":"1.29.2","filebeatenabled":"true","maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"}}}},"provider":"ae47cfb1-d32c-4a26-a8f5-e5d587bd48db","start":true}}
filebeat_1             | 2021-12-02T16:37:11.698Z	DEBUG	[autodiscover]	autodiscover/autodiscover.go:193	Generated config: {
filebeat_1             |   "format": "docker",
filebeat_1             |   "paths": [
filebeat_1             |     "/var/lib/docker/containers/70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f/*.log"
filebeat_1             |   ],
filebeat_1             |   "processors": [
filebeat_1             |     {
filebeat_1             |       "add_docker_metadata": null
filebeat_1             |     },
filebeat_1             |     {
filebeat_1             |       "decode_json_fields": {
filebeat_1             |         "fields": [
filebeat_1             |           "message"
filebeat_1             |         ],
filebeat_1             |         "overwrite_keys": true,
filebeat_1             |         "target": "",
filebeat_1             |         "when": {
filebeat_1             |           "equals": {
filebeat_1             |             "docker": {
filebeat_1             |               "container": {
filebeat_1             |                 "labels": {
filebeat_1             |                   "decodeLogEventToJsonObject": "true"
filebeat_1             |                 }
filebeat_1             |               }
filebeat_1             |             }
filebeat_1             |           }
filebeat_1             |         }
filebeat_1             |       }
filebeat_1             |     }
filebeat_1             |   ],
filebeat_1             |   "type": "container"
filebeat_1             | }
filebeat_1             | 2021-12-02T16:37:11.730Z	INFO	[input]	log/input.go:164	Configured paths: [/var/lib/docker/containers/70bdc0fabd6b4cfbee1cc5d76b19ece3495e2acd11b6a7b175a99342ab47067f/*.log]	{"input_id": "a5cd6039-021c-47c9-9048-d8bd8746e3af"}
filebeat_1             | 2021-12-02T16:37:11.732Z	DEBUG	[input]	log/input.go:215	Start next scan	{"input_id": "a5cd6039-021c-47c9-9048-d8bd8746e3af"}
filebeat_1             | 2021-12-02T16:37:11.732Z	DEBUG	[input]	log/input.go:279	input states cleaned up. Before: 0, After: 0, Pending: 0	{"input_id": "a5cd6039-021c-47c9-9048-d8bd8746e3af"}

From kibana->index management i see the index created with filebeat setup, but with 0 docs count.

Any help appreciated, thanks in advance.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.