Filebeat 8.14 not sending data to Elasticsearch self-hosted

Elastic Stack Beats
,
1

Problem

I want to test our migration from Elastic-Stack 7 to 8. I have set up an elasticserch and kibana server. This is working fine. Now, I try to send docker container logs from a filebeat 8 instance to the elasticsearch 8 test-instance. This is not working because filebeat not sending any data to the server.

Configs

filebeat.yml:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

filebeat.inputs:
- type: container
  enabled: true
  paths:
    - '/var/log/containers/*.log'

processors:
  - add_docker_metadata: ~
  - add_fields:
      target: ''
      fields:
        environment: "test"
        stack: "test-stack"
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: 1
      target: ""
      overwrite_keys: true
      add_error_key: true

  - dissect:
      when:
        has_fields: ['decode_json_fields.error']
      tokenizer: "[%{timestamp}] %{log_level} %{thread} [%{transaction}] %{message}"
      field: "message"
      target_prefix: "a"
      append_separator: " "
      overwrite_keys: false

output.elasticsearch:
  hosts: 'elastic.100.82.58.68.nip.io:80'
  protocol: http
  path: /elasticsearch
  username: 'elastic'
  password: 'password'
  ssl.enabled: false

setup.kibana:
  host: "elastic.100.82.58.68.nip.io:80"
  username: "elastic"
  password: "password"
  protocol: http
  ssl.enabled: false

logging:
  level: debug

Logs

Config appears to be okay:

root@c78de879e9e9:/usr/share/filebeat# filebeat test config
Config OK
root@c78de879e9e9:/usr/share/filebeat#

Connect appears also be okay:

elasticsearch: http://elastic.100.82.58.68.nip.io:80/elasticsearch...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 100.82.58.68
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 8.14.0

filebeat setup was done successfully:

root@c78de879e9e9:/usr/share/filebeat# filebeat setup
Overwriting lifecycle policy is disabled. Set `setup.ilm.overwrite: true` to overwrite.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Loaded Ingest pipelines
root@c78de879e9e9:/usr/share/filebeat#

This is the startup log of filebeat:

{"log.level":"info","@timestamp":"2024-06-06T13:31:58.711Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":816},"message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.711Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).loadMeta","file.name":"instance/beat.go","file.line":935},"message":"Beat metadata path: /usr/share/filebeat/data/meta.json","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.711Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure","file.name":"instance/beat.go","file.line":824},"message":"Beat ID: 49d44afd-020a-426a-badc-77d7a4cbaf72","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.712Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.NewClient","file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.723Z","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors/add_docker_metadata.buildDockerMetadataProcessor","file.name":"add_docker_metadata/add_docker_metadata.go","file.line":92},"message":"add_docker_metadata: docker environment detected","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.723Z","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).Start","file.name":"docker/watcher.go","file.line":213},"message":"Start docker containers scanner","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.723Z","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).listContainers","file.name":"docker/watcher.go","file.line":375},"message":"List containers","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.724Z","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).listContainers","file.name":"docker/watcher.go","file.line":399},"message":"Inspect container 8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"add_docker_metadata","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).watch.func1","file.name":"docker/watcher.go","file.line":266},"message":"Fetching events since 2024-06-06 13:31:58.726610869 +0000 UTC m=+0.058915270","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"conditions","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/conditions.NewCondition","file.name":"conditions/conditions.go","file.line":98},"message":"New condition has_fields: [decode_json_fields.error]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"processors","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/processors.New","file.name":"processors/processor.go","file.line":114},"message":"Generated new processors: add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.parent.pid]], add_fields={\"environment\":\"test\",\"stack\":\"tools-stack\"}, decode_json_fields=message, dissect=[%{timestamp}] %{log_level} %{thread} [%{transaction}] %{message},field=message,target_prefix=a, condition=has_fields: [decode_json_fields.error]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"add_docker_metadata.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000eec0e0 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"add_docker_metadata.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000ee4380 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"add_docker_metadata.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000ee43f0 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.726Z","log.logger":"seccomp","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter","file.name":"seccomp/seccomp.go","file.line":118},"message":"Loading syscall filter","service.name":"filebeat","seccomp_filter":{"no_new_privs":true,"flag":"tsync","policy":{"default_action":"allow","syscalls":[{"names":["execveat"],"names_with_args":null,"action":"errno"}]}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.727Z","log.logger":"seccomp","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/common/seccomp.loadFilter","file.name":"seccomp/seccomp.go","file.line":125},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.727Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1370},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"49d44afd-020a-426a-badc-77d7a4cbaf72"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.727Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1379},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"de52d1434ea3dff96953a59a18d44e456a98bd2f","libbeat":"8.14.0","time":"2024-05-31T15:23:37.000Z","version":"8.14.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.727Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1382},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"arm64","max_procs":8,"version":"go1.21.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.728Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1388},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"aarch64","native_architecture":"aarch64\n","boot_time":"2024-05-24T16:29:09Z","containerized":false,"name":"c78de879e9e9","ip":["127.0.0.1","172.17.0.2"],"kernel_version":"6.5.0-28-generic","mac":["02:42:ac:11:00:02"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.728Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo","file.name":"instance/beat.go","file.line":1417},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/filebeat","exe":"/usr/share/filebeat/filebeat","name":"filebeat","pid":264,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-06-06T13:31:58.510Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.728Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":339},"message":"Setup Beat: filebeat; Version: 8.14.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.728Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater","file.name":"instance/beat.go","file.line":367},"message":"Initializing output plugins","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.729Z","log.logger":"esclientleg","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection","file.name":"eslegclient/connection.go","file.line":122},"message":"elasticsearch url: http://elastic.100.82.58.68.nip.io:80/elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.729Z","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).run","file.name":"pipeline/consumer.go","file.line":110},"message":"start pipeline event consumer","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.729Z","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*queueReader).run","file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.729Z","log.logger":"publisher","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings","file.name":"pipeline/module.go","file.line":105},"message":"Beat name: c78de879e9e9","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.729Z","log.logger":"modules","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry","file.name":"fileset/modules.go","file.line":136},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.730Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop","file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.731Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch","file.name":"instance/beat.go","file.line":525},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.731Z","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile(/usr/share/filebeat/data/registry) -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.731Z","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile() -> false","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.731Z","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isDir","file.name":"registrar/migrate.go","file.line":280},"message":"isDir(/usr/share/filebeat/data/registry/filebeat) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.731Z","log.logger":"test","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.isFile","file.name":"registrar/migrate.go","file.line":287},"message":"isFile(/usr/share/filebeat/data/registry/filebeat/meta.json) -> true","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.731Z","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Migrator).Run","file.name":"registrar/migrate.go","file.line":82},"message":"Registry type '1' found","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.731Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore","file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/statestore/backend/memlog.openStore","file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/x-pack/filebeat/input/shipper.NewInputManager","file.name":"shipper/input.go","file.line":55},"message":"creating new InputManager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).loadStates","file.name":"registrar/registrar.go","file.line":107},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"cfgwarn","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.NewInput.func1","file.name":"log/input.go","file.line":90},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*config).resolveRecursiveGlobs","file.name":"log/config.go","file.line":207},"message":"recursive glob enabled","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).loadStates","file.name":"log/input.go","file.line":188},"message":"exclude_files: []. Number of states: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).loadStates","file.name":"log/input.go","file.line":221},"message":"input with previous states loaded: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.NewInput","file.name":"log/input.go","file.line":174},"message":"Configured paths: [/var/log/containers/*.log]","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).startInput","file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 15395659083897266066)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.733Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check","file.name":"cfgfile/reload.go","file.line":131},"message":"Checking module configs from: /usr/share/filebeat/modules.d/*.yml","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"registrar","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/registrar.(*Registrar).Run","file.name":"registrar/registrar.go","file.line":138},"message":"Starting Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Check","file.name":"cfgfile/reload.go","file.line":145},"message":"Number of module configs found: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"crawler","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/beater.(*crawler).Start","file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.734Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.NewClient","file.name":"docker/client.go","file.line":49},"message":"Docker client will negotiate the API version on the first request.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.734Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.736Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":193},"message":"Scan for new config files","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.736Z","log.logger":"cfgfile","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":212},"message":"Number of module configs found: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.736Z","log.logger":"reload","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload","file.name":"cfgfile/list.go","file.line":92},"message":"Starting reload procedure, current runners: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.736Z","log.logger":"reload","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload","file.name":"cfgfile/list.go","file.line":110},"message":"Start list: 0, Stop list: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.736Z","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run","file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.743Z","log.logger":"modules","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry","file.name":"fileset/modules.go","file.line":136},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.743Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).Start","file.name":"docker/watcher.go","file.line":213},"message":"Start docker containers scanner","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.743Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).listContainers","file.name":"docker/watcher.go","file.line":375},"message":"List containers","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.745Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).listContainers","file.name":"docker/watcher.go","file.line":399},"message":"Inspect container 8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.NewAutodiscover","file.name":"autodiscover/autodiscover.go","file.line":90},"message":"Configured autodiscover provider: docker","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).Start","file.name":"autodiscover/autodiscover.go","file.line":114},"message":"Starting autodiscover manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"docker.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000ee4e00 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"docker.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000f0a850 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"docker.bus-docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[container:0x4000f0a8c0 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"docker","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"docker","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/docker.(*watcher).watch.func1","file.name":"docker/watcher.go","file.line":266},"message":"Fetching events since 2024-06-06 13:31:58.746755242 +0000 UTC m=+0.079059644","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"hints.builder","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints.(*logHints).CreateConfig","file.name":"hints/logs.go","file.line":201},"message":"hints.builder%!(EXTRA string=generated config %+v of logHints %+v, *config.C=&{{<nil> } <nil> 0x40012864c0}, *hints.logHints=&{0x4000cd8390 0x4000df25a0 0x4000cef2e0})","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.746Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'id'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"hints.builder","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints.(*logHints).CreateConfig","file.name":"hints/logs.go","file.line":201},"message":"hints.builder%!(EXTRA string=generated config %+v of logHints %+v, *config.C=&{{<nil> } <nil> 0x40012868a0}, *hints.logHints=&{0x4000cd8390 0x4000df25a0 0x4000cef2e0})","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover.bus-filebeat","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[config:[] container:{\"id\":\"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede\",\"image\":{\"name\":\"nginx\"},\"labels\":{\"maintainer\":\"NGINX Docker Maintainers \\u003cdocker-maint@nginx.com\\u003e\"},\"name\":\"strange_black\"} docker:{\"container\":{\"id\":\"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede\",\"image\":\"nginx\",\"labels\":{\"maintainer\":\"NGINX Docker Maintainers \\u003cdocker-maint@nginx.com\\u003e\"},\"name\":\"strange_black\"}} host:172.17.0.3 id:0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede meta:{\"container\":{\"id\":\"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede\",\"image\":{\"name\":\"nginx\"},\"name\":\"strange_black\"},\"docker\":{\"container\":{\"labels\":{\"maintainer\":\"NGINX Docker Maintainers \\u003cdocker-maint@nginx.com\\u003e\"}}}} provider:cab80e96-b901-40dd-a938-4b61c7f5e4d3 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"hints.builder","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints.(*logHints).CreateConfig","file.name":"hints/logs.go","file.line":201},"message":"hints.builder%!(EXTRA string=generated config %+v of logHints %+v, *config.C=&{{<nil> } <nil> 0x4001286c80}, *hints.logHints=&{0x4000cd8390 0x4000df25a0 0x4000cef2e0})","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover.bus-filebeat","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[config:[] container:{\"id\":\"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5\",\"image\":{\"name\":\"nicolaka/netshoot\"},\"labels\":{},\"name\":\"vigilant_napier\"} docker:{\"container\":{\"id\":\"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5\",\"image\":\"nicolaka/netshoot\",\"labels\":{},\"name\":\"vigilant_napier\"}} host:c78de879e9e9 id:8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5 meta:{\"container\":{\"id\":\"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5\",\"image\":{\"name\":\"nicolaka/netshoot\"},\"name\":\"vigilant_napier\"},\"docker\":{\"container\":{\"labels\":{}}}} provider:cab80e96-b901-40dd-a938-4b61c7f5e4d3 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"hints.builder","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/autodiscover/builder/hints.(*logHints).CreateConfig","file.name":"hints/logs.go","file.line":201},"message":"hints.builder%!(EXTRA string=generated config %+v of logHints %+v, *config.C=&{{<nil> } <nil> 0x4001287020}, *hints.logHints=&{0x4000cd8390 0x4000df25a0 0x4000cef2e0})","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover/template.ApplyConfigTemplate","file.name":"template/config.go","file.line":157},"message":"Configuration template cannot be resolved: field 'data.kubernetes.container.id' not available in event or environment accessing 'paths'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover.bus-filebeat","log.origin":{"function":"github.com/elastic/elastic-agent-autodiscover/bus.(*bus).Publish","file.name":"bus/bus.go","file.line":94},"message":"map[config:[] container:{\"id\":\"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771\",\"image\":{\"name\":\"docker.elastic.co/beats/filebeat:8.14.0\"},\"labels\":{\"description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"io\":{\"k8s\":{\"description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"display-name\":\"Filebeat image\"}},\"license\":\"Elastic License\",\"maintainer\":\"infra@elastic.co\",\"name\":\"filebeat\",\"org\":{\"label-schema\":{\"build-date\":\"2024-05-31T15:22:45Z\",\"license\":\"Elastic License\",\"name\":\"filebeat\",\"schema-version\":\"1.0\",\"url\":\"https://www.elastic.co/beats/filebeat\",\"vcs-ref\":\"de52d1434ea3dff96953a59a18d44e456a98bd2f\",\"vcs-url\":\"github.com/elastic/beats/v7\",\"vendor\":\"Elastic\",\"version\":\"8.14.0\"},\"opencontainers\":{\"image\":{\"created\":\"2024-05-31T15:22:45Z\",\"licenses\":\"Elastic License\",\"ref\":{\"name\":\"ubuntu\"},\"title\":\"Filebeat\",\"vendor\":\"Elastic\",\"version\":\"20.04\"}}},\"release\":\"1\",\"summary\":\"filebeat\",\"url\":\"https://www.elastic.co/beats/filebeat\",\"vendor\":\"Elastic\",\"version\":\"8.14.0\"},\"name\":\"filebeat\"} docker:{\"container\":{\"id\":\"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771\",\"image\":\"docker.elastic.co/beats/filebeat:8.14.0\",\"labels\":{\"description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"io\":{\"k8s\":{\"description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"display-name\":\"Filebeat image\"}},\"license\":\"Elastic License\",\"maintainer\":\"infra@elastic.co\",\"name\":\"filebeat\",\"org\":{\"label-schema\":{\"build-date\":\"2024-05-31T15:22:45Z\",\"license\":\"Elastic License\",\"name\":\"filebeat\",\"schema-version\":\"1.0\",\"url\":\"https://www.elastic.co/beats/filebeat\",\"vcs-ref\":\"de52d1434ea3dff96953a59a18d44e456a98bd2f\",\"vcs-url\":\"github.com/elastic/beats/v7\",\"vendor\":\"Elastic\",\"version\":\"8.14.0\"},\"opencontainers\":{\"image\":{\"created\":\"2024-05-31T15:22:45Z\",\"licenses\":\"Elastic License\",\"ref\":{\"name\":\"ubuntu\"},\"title\":\"Filebeat\",\"vendor\":\"Elastic\",\"version\":\"20.04\"}}},\"release\":\"1\",\"summary\":\"filebeat\",\"url\":\"https://www.elastic.co/beats/filebeat\",\"vendor\":\"Elastic\",\"version\":\"8.14.0\"},\"name\":\"filebeat\"}} host:172.17.0.2 id:c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771 meta:{\"container\":{\"id\":\"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771\",\"image\":{\"name\":\"docker.elastic.co/beats/filebeat:8.14.0\"},\"name\":\"filebeat\"},\"docker\":{\"container\":{\"labels\":{\"description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"io_k8s_description\":\"Filebeat sends log files to Logstash or directly to Elasticsearch.\",\"io_k8s_display-name\":\"Filebeat image\",\"license\":\"Elastic License\",\"maintainer\":\"infra@elastic.co\",\"name\":\"filebeat\",\"org_label-schema_build-date\":\"2024-05-31T15:22:45Z\",\"org_label-schema_license\":\"Elastic License\",\"org_label-schema_name\":\"filebeat\",\"org_label-schema_schema-version\":\"1.0\",\"org_label-schema_url\":\"https://www.elastic.co/beats/filebeat\",\"org_label-schema_vcs-ref\":\"de52d1434ea3dff96953a59a18d44e456a98bd2f\",\"org_label-schema_vcs-url\":\"github.com/elastic/beats/v7\",\"org_label-schema_vendor\":\"Elastic\",\"org_label-schema_version\":\"8.14.0\",\"org_opencontainers_image_created\":\"2024-05-31T15:22:45Z\",\"org_opencontainers_image_licenses\":\"Elastic License\",\"org_opencontainers_image_ref_name\":\"ubuntu\",\"org_opencontainers_image_title\":\"Filebeat\",\"org_opencontainers_image_vendor\":\"Elastic\",\"org_opencontainers_image_version\":\"20.04\",\"release\":\"1\",\"summary\":\"filebeat\",\"url\":\"https://www.elastic.co/beats/filebeat\",\"vendor\":\"Elastic\",\"version\":\"8.14.0\"}}}} provider:cab80e96-b901-40dd-a938-4b61c7f5e4d3 start:true]","service.name":"filebeat","elastic-agent-autodiscover.bus":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart","file.name":"autodiscover/autodiscover.go","file.line":189},"message":"Got a start event.","service.name":"filebeat","autodiscover.event":{"config":[],"container":{"id":"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede","image":{"name":"nginx"},"labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"name":"strange_black"},"docker":{"container":{"id":"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede","image":"nginx","labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"},"name":"strange_black"}},"host":"172.17.0.3","id":"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede","meta":{"container":{"id":"0524139acecee907409640b24d990990e46d68532fa759d7f3c48e3a5eedeede","image":{"name":"nginx"},"name":"strange_black"},"docker":{"container":{"labels":{"maintainer":"NGINX Docker Maintainers <docker-maint@nginx.com>"}}}},"provider":"cab80e96-b901-40dd-a938-4b61c7f5e4d3","start":true},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).getMeta","file.name":"autodiscover/autodiscover.go","file.line":297},"message":"Got a meta field in the event","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart","file.name":"autodiscover/autodiscover.go","file.line":189},"message":"Got a start event.","service.name":"filebeat","autodiscover.event":{"config":[],"container":{"id":"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","image":{"name":"nicolaka/netshoot"},"labels":{},"name":"vigilant_napier"},"docker":{"container":{"id":"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","image":"nicolaka/netshoot","labels":{},"name":"vigilant_napier"}},"host":"c78de879e9e9","id":"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","meta":{"container":{"id":"8b4e10f06d470298b1f135b202c3d8d91cd46db829900bab0747bc25512737b5","image":{"name":"nicolaka/netshoot"},"name":"vigilant_napier"},"docker":{"container":{"labels":{}}}},"provider":"cab80e96-b901-40dd-a938-4b61c7f5e4d3","start":true},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).getMeta","file.name":"autodiscover/autodiscover.go","file.line":297},"message":"Got a meta field in the event","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).handleStart","file.name":"autodiscover/autodiscover.go","file.line":189},"message":"Got a start event.","service.name":"filebeat","autodiscover.event":{"config":[],"container":{"id":"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771","image":{"name":"docker.elastic.co/beats/filebeat:8.14.0"},"labels":{"description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","io":{"k8s":{"description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","display-name":"Filebeat image"}},"license":"Elastic License","maintainer":"infra@elastic.co","name":"filebeat","org":{"label-schema":{"build-date":"2024-05-31T15:22:45Z","license":"Elastic License","name":"filebeat","schema-version":"1.0","url":"https://www.elastic.co/beats/filebeat","vcs-ref":"de52d1434ea3dff96953a59a18d44e456a98bd2f","vcs-url":"github.com/elastic/beats/v7","vendor":"Elastic","version":"8.14.0"},"opencontainers":{"image":{"created":"2024-05-31T15:22:45Z","licenses":"Elastic License","ref":{"name":"ubuntu"},"title":"Filebeat","vendor":"Elastic","version":"20.04"}}},"release":"1","summary":"filebeat","url":"https://www.elastic.co/beats/filebeat","vendor":"Elastic","version":"8.14.0"},"name":"filebeat"},"docker":{"container":{"id":"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771","image":"docker.elastic.co/beats/filebeat:8.14.0","labels":{"description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","io":{"k8s":{"description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","display-name":"Filebeat image"}},"license":"Elastic License","maintainer":"infra@elastic.co","name":"filebeat","org":{"label-schema":{"build-date":"2024-05-31T15:22:45Z","license":"Elastic License","name":"filebeat","schema-version":"1.0","url":"https://www.elastic.co/beats/filebeat","vcs-ref":"de52d1434ea3dff96953a59a18d44e456a98bd2f","vcs-url":"github.com/elastic/beats/v7","vendor":"Elastic","version":"8.14.0"},"opencontainers":{"image":{"created":"2024-05-31T15:22:45Z","licenses":"Elastic License","ref":{"name":"ubuntu"},"title":"Filebeat","vendor":"Elastic","version":"20.04"}}},"release":"1","summary":"filebeat","url":"https://www.elastic.co/beats/filebeat","vendor":"Elastic","version":"8.14.0"},"name":"filebeat"}},"host":"172.17.0.2","id":"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771","meta":{"container":{"id":"c78de879e9e9ef54d3496814249d16e487d7b5e39ec186673bc91e4cc92c9771","image":{"name":"docker.elastic.co/beats/filebeat:8.14.0"},"name":"filebeat"},"docker":{"container":{"labels":{"description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","io_k8s_description":"Filebeat sends log files to Logstash or directly to Elasticsearch.","io_k8s_display-name":"Filebeat image","license":"Elastic License","maintainer":"infra@elastic.co","name":"filebeat","org_label-schema_build-date":"2024-05-31T15:22:45Z","org_label-schema_license":"Elastic License","org_label-schema_name":"filebeat","org_label-schema_schema-version":"1.0","org_label-schema_url":"https://www.elastic.co/beats/filebeat","org_label-schema_vcs-ref":"de52d1434ea3dff96953a59a18d44e456a98bd2f","org_label-schema_vcs-url":"github.com/elastic/beats/v7","org_label-schema_vendor":"Elastic","org_label-schema_version":"8.14.0","org_opencontainers_image_created":"2024-05-31T15:22:45Z","org_opencontainers_image_licenses":"Elastic License","org_opencontainers_image_ref_name":"ubuntu","org_opencontainers_image_title":"Filebeat","org_opencontainers_image_vendor":"Elastic","org_opencontainers_image_version":"20.04","release":"1","summary":"filebeat","url":"https://www.elastic.co/beats/filebeat","vendor":"Elastic","version":"8.14.0"}}}},"provider":"cab80e96-b901-40dd-a938-4b61c7f5e4d3","start":true},"ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:31:58.747Z","log.logger":"autodiscover","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/autodiscover.(*Autodiscover).getMeta","file.name":"autodiscover/autodiscover.go","file.line":297},"message":"Got a meta field in the event","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:08.734Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":141},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:08.735Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:08.735Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:18.735Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":141},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:18.735Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:18.735Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:28.737Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":141},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:28.737Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:28.737Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-06-06T13:32:28.747Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"id":"/"},"memory":{"id":"/","mem":{"usage":{"bytes":38494208}}}},"cpu":{"system":{"ticks":70,"time":{"ms":70}},"total":{"ticks":110,"time":{"ms":110},"value":110},"user":{"ticks":40,"time":{"ms":40}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":11},"info":{"ephemeral_id":"57e9af61-132e-4572-83c1-28b6b45acab9","name":"filebeat","uptime":{"ms":30046},"version":"8.14.0"},"memstats":{"gc_next":30253160,"memory_alloc":27289952,"memory_sys":40998152,"memory_total":37048080,"rss":102498304},"runtime":{"goroutines":41}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch","write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.47,"15":0.27,"5":0.34,"norm":{"1":0.0588,"15":0.0338,"5":0.0425}}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:38.738Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input.(*Runner).Run","file.name":"input/input.go","file.line":141},"message":"Run input","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:38.739Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).Run","file.name":"log/input.go","file.line":228},"message":"Start next scan","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2024-06-06T13:32:38.739Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).cleanupStates","file.name":"log/input.go","file.line":292},"message":"input states cleaned up. Before: 0, After: 0, Pending: 0","service.name":"filebeat","input_id":"e1bd00c1-aee2-4813-ae1e-135c14adbbfd","ecs.version":"1.6.0"}

The index template is created

name            index_patterns    order version composed_of
filebeat-8.14.0 [filebeat-8.14.0] 150           []

The index is also there but no docuemnts in the index:

health status index                                 uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   .ds-filebeat-8.14.0-2024.06.06-000001 BFUSP_2xTXiYOq7P2x1IgA   1   1          0            0       249b           249b         249b

What i've done so far

I've deleted all data folders and started with the Elastic-Stack set up from scratch. Nothing helps so far.

Thanks for your help.

2

Hi @menzbua, Welcome to the Elastic communtiy.

There are breaking changes between major version (7.x to 8.x). Also when you say you have deleted all data folder, Is it elasticsearch's data folder ? Or you have re installed Elasticsearch, kibana and filebeat with same version ?

3

I have the version 8 set up in parallel. Our prod environment is running on other machines with version 7 at the moment. I can play around with the 8 set up and test our configurations.
So I've set up elasticsearch and kibana in version 8.14 on our test environment. I want that filebeat 8.14, that is also installed on another test server, to send data to the elasticsearch 8 instance.

4

Hi @menzbua Welcome to the community.

From your debug logs filebeat has no files open / events that were read and sent to elastic.

{"log.level":"info","@timestamp":"2024-06-06T13:32:28.747Z","log.logger":"monitoring",......,"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch","write":{"latency":{"histogram":{"count":0,"max":0,"mean":0,"median":0,"min":0,"p75":0,"p95":0,"p99":0,"p999":0,"stddev":0}}}},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":3200}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.47,"15":0.27,"5":0.34,"norm":{"1":0.0588,"15":0.0338,"5":0.0425}}}},"ecs.version":"1.6.0"}}

I.e There was nothing to read....

5

Okay, the docker autodiscovery appears to be different with version >8.12 (see: Filebeat Docker Autodiscovery stopped working when I upgraded to 8.12.0 - #10 by Michalis_Katsoulis).
I've modified my config:

filebeat.autodiscover:
  providers:
    - type: docker
      hints.default_config:
        paths:
          - /var/lib/docker/containers/${data.container.id}/*.log
        type: container
      hints.enabled: true

Now it seems to be working.

1 Like