I have Elasticsearch and Kibana working, that was the easy part.
I can ingest files directly from Kibana, no problems they get parsed and handled without issue.
I am now trying to configure filebeat to pull in files from disk.
The logs seem to indicate no problem connecting to elasticsearch, and I can see an index for the filebeat connection to Kibana.
yellow open .ds-filebeat-8.4.2-2022.09.29-000001 vgkrXtxrQKalFguxA5r-Aw 1 1 229160 0 65.9mb 65.9mb
However the files are not being loaded by filebeat.
I can see that filebeat notices the log file:
Sep 29 12:35:42 optimusprime filebeat[4182105]: {"log.level":"info","@timestamp":"2022-09-29T12:35:42.055-0400","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"NPCStream","source_file":"filestream::NPCStream::native::536871118-64770","path":"/home/logs/xaa.log","state-id":"native::536871118-64770","ecs.version":"1.6.0"}
but it never gets parsed and ingested.
I took some sample json items from the internet hoping it would help here but it didn't
# filestream is an input for collecting log messages from files.
- type: filestream
id: NPCStream
enabled: true
paths:
- /home/logs/*.log
json.keys_under_root: true
json.overwrite_keys: true
json.add_error_key: true
json.expand_keys: true
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]