While ingesting a log file from source system to Kibana through Filebeat, getting "End Of File reached" message in Filebeat logs

[cadrija]

Hi experts!

I am new to elastic. I have installed ELK (8.4) on a Ubuntu system (suppose u.u.u.u). Now I am trying to fetch/ingest a log file from a windows system (suppose w.w.w.w).

Steps I followed leaning from tutorials:-

1. Installed Filebeat (8.4) on windows system (w.w.w.w)
2. Set the connection information in filebeat.yml for Elasticsearch & Kibana.
3. As I want the log file that I kept in the Documents folder (for testing purpose). Hence I configured the input manually in filebeat.yml
4. I ran .\filebeat.exe setup -e
5. I ran Start-Service filebeat
6. In the side navigation, clicked Discover. To see Filebeat data, with the predefined filebeat-* index pattern selected. But no results, tried expanding time till last 1 year also.
7. Turned the filebeat log level up to debug. Getting "End of file reached" message over & over.

My filebeat.yml file's input
My log file location - C:\Users\adri\Documents\XYZ_logs\XYZ.log
I do not have any idea what id field expects, hence kept 'xyz'

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: xyz

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - C:\Users\adri\Documents\XYZ_logs\XYZ.log

After Filebeat finds my log file, this is the part of logs that prints over & over.

{"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:01.448Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:07.460Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:07.461Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:11.452Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:18.743Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:18.744Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-14T12:46:20.269Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":1531,"time":{"ms":1531},"value":1531},"user":{"ticks":1406,"time":{"ms":1406}}},"info":{"ephemeral_id":"02ca6067-8f48-493d-8172-3f916076f7ec","name":"filebeat","uptime":{"ms":51335},"version":"8.4.3"},"memstats":{"gc_next":19892080,"memory_alloc":11033600,"memory_sys":27429896,"memory_total":58642840,"rss":58712064},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"handles":{"open":230}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:21.877Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}