Filebeat not ingesting json based log files to elastic search

I have Elasticsearch and Kibana working, that was the easy part.
I can ingest files directly from Kibana, no problems they get parsed and handled without issue.

I am now trying to configure filebeat to pull in files from disk.

The logs seem to indicate no problem connecting to elasticsearch, and I can see an index for the filebeat connection to Kibana.

yellow open .ds-filebeat-8.4.2-2022.09.29-000001 vgkrXtxrQKalFguxA5r-Aw 1 1 229160 0 65.9mb 65.9mb

However the files are not being loaded by filebeat.

I can see that filebeat notices the log file:
Sep 29 12:35:42 optimusprime filebeat[4182105]: {"log.level":"info","@timestamp":"2022-09-29T12:35:42.055-0400","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"NPCStream","source_file":"filestream::NPCStream::native::536871118-64770","path":"/home/logs/xaa.log","state-id":"native::536871118-64770","ecs.version":"1.6.0"}

but it never gets parsed and ingested.
I took some sample json items from the internet hoping it would help here but it didn't

# filestream is an input for collecting log messages from files.
- type: filestream
  id: NPCStream
  enabled: true
  paths:
    - /home/logs/*.log

  json.keys_under_root: true
  json.overwrite_keys: true
  json.add_error_key: true
  json.expand_keys: true
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

Might be that file is already read and exist in the registry database.
Check /var/lib/filebeat. You can also set as a parameter: filebeat.registry.path: registry

If still is not working, change logging to the debug mode.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.