How to ingest a log file from source system to Kibana through Filebeat

cadrija · September 30, 2022, 10:30am

Hi experts!

I am new to elastic. I have installed ELK (7.17.6) on a Ubuntu system (suppose u.u.u.u). Now I am trying to fetch/ingest a log file from a windows system (suppose w.w.w.w).

Steps I followed leaning from tutorials:-

Installed Filebeat (7.17.6) on windows system (w.w.w.w)
Set the connection information in filebeat.yml for Elasticsearch & Kibana.
As I want the log file that I kept in the Documents folder (for testing purpose). Hence I configured the input manually in filebeat.yml
I ran .\filebeat.exe setup -e
I ran Start-Service filebeat
In the side navigation, clicked Discover. To see Filebeat data, with the predefined filebeat-* index pattern selected. But no results, tried expanding time till last 1 year also.

My filebeat.yml file's input
My log file location - C:\Users\mida\Documents\XYZ logs\xyz.log
I do not have any idea what id field expects, hence kept 'xyz'

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: xyz

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - C:\Users\mida\Documents\XYZ logs\xyz.log

weltenwort · October 4, 2022, 11:57am

Hi @cadrija,

your configuration looks good so far, but this line suggests the input is still disabled. Was this a copy/paste error?

cadrija · October 10, 2022, 4:56am

@weltenwort Thanks for your response.
Sorry for my late reply.
I am not aware about the field I followed some tutorial and there it was not mentioned to update/change so I left it as it was.
Now I updated it to enabled: false. But still not getting the logs in Discover.

weltenwort · October 10, 2022, 11:49am

Just to be clear: it should be enabled: true to enable this input. Can you double-check and provide the filebeat log output if it still doesn't work?

cadrija · October 10, 2022, 2:10pm

@weltenwort I am sorry, it's my typing mistake. I updated it to enabled: true.

The current filebeat inputs section:

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: xyz

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*
    - C:\Users\mida\Documents\XYZ logs\xyz.log

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

weltenwort · October 10, 2022, 2:37pm

That looks fine so far. Can you provide the log output of filebeat so we can see whether it logs any errors reading the file or connecting to Elasticsearch?

cadrija · October 11, 2022, 4:53am

@weltenwort Today it's showing hits in Kibana from filebeat-* but the logs are cloud metadata not the one from the log file that I intend to pull i.e. C:\Users\mida\Documents\XYZ_logs\xyz.log (I added an underscore in the folder name XYZ_logs, just to be sure that the space in the name is not creating any problem)

I checked the logs folder & found only 1 line entry for the mentioned id and no errors.

2022-10-11T06:46:26.589Z	INFO	[input.filestream]	compat/compat.go:113	Input 'filestream' starting	{"id": "xyz"}

weltenwort · October 11, 2022, 11:35am

Are you looking to ingest the log lines as they are continuously written by another process or is that a "finished" log file that you want to ingest retroactively?

The background of that question is, that filebeat keeps track of its last position in the file as to not ingest lines twice. That information is stored in the data directory. Maybe it already considers the lines read during your experimentation? As I see from your paths you're running this on Windows, which comes with extra caveats regarding file truncation and identity. Maybe Log rotation results in lost or duplicate events | Filebeat Reference [8.4] | Elastic applies here?

Sorry if this feels like fishing in the dark, but I don't know enough about your infrastructure to narrow down the possible causes any other way.

cadrija · October 11, 2022, 12:41pm

@weltenwort Thanks for the response.
For time being I am testing with a "finished" log file.
If this is success then I will get the approval to try my hands on live log file that are continuously written by another process.

weltenwort · October 11, 2022, 2:30pm

So maybe filebeat already marked your file as "read" in its registry (see How does Filebeat keep the state of files?). You could stop filebeat and delete the files in the data directory of filebeat to clear out its state. This assumes that the filebeat instance doesn't keep track of any other log files that would be affected, though.

cadrija · October 12, 2022, 5:09am

@weltenwort Thank you for this input. Let me try this out.

cadrija · October 13, 2022, 6:16am

@weltenwort Tried with different log files, but still no luck

weltenwort · October 13, 2022, 12:15pm

hm The next thing I would do is to turn the filebeat log level up to debug and take a look at its log output to see if it reports anything that might be related. Would you be able to provide that log output after a restart of filebeat?

cadrija · October 17, 2022, 6:26am

@weltenwort Thanks for the guidance.
After Filebeat finds my log file, this is the part of logs that prints over & over.

{"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:01.448Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:07.460Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:07.461Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:11.452Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:18.743Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:18.744Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-10-14T12:46:20.269Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":1531,"time":{"ms":1531},"value":1531},"user":{"ticks":1406,"time":{"ms":1406}}},"info":{"ephemeral_id":"02ca6067-8f48-493d-8172-3f916076f7ec","name":"filebeat","uptime":{"ms":51335},"version":"8.4.3"},"memstats":{"gc_next":19892080,"memory_alloc":11033600,"memory_sys":27429896,"memory_total":58642840,"rss":58712064},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"handles":{"open":230}}},"ecs.version":"1.6.0"}}
{"log.level":"debug","@timestamp":"2022-10-14T12:46:21.877Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"}

weltenwort · November 8, 2022, 11:52am

Looks like it "knows" the file has not changed. Did you delete the cursor information (stored in the data directory)? After deleting the state it should re-index the file once and then do nothing until the file changes.

cadrija · November 10, 2022, 5:50am

@weltenwort Thank for the response.

I uninstalled & deleted all traces of 7.x Filebeat then installed 8.x Filebeat but still getting same message in logs.

system · December 8, 2022, 5:51am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.