# How to ingest a log file from source system to Kibana through Filebeat **URL:** \ **Category:** Logs\ **Created:** [September 30, 2022, 10:30am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544 "2022-09-30T10:30:09Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [September 30, 2022, 10:30am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/1 "2022-09-30T10:30:10Z")
Hi experts! I am new to elastic. I have installed ELK (7.17.6) on a Ubuntu system (suppose u.u.u.u). Now I am trying to fetch/ingest a log file from a windows system (suppose w.w.w.w). Steps I followed leaning from tutorials:- 1. Installed Filebeat (7.17.6) on windows system (w.w.w.w) 2. Set the connection information in `filebeat.yml` for Elasticsearch & Kibana. 3. As I want the log file that I kept in the Documents folder (for testing purpose). Hence I configured the input manually in `filebeat.yml` 4. I ran .\filebeat.exe setup -e 5. I ran Start-Service filebeat 6. In the side navigation, clicked **Discover**. To see Filebeat data, with the predefined `filebeat-*` index pattern selected. But no results, tried expanding time till last 1 year also. **My filebeat.yml file's input** My log file location - C:\Users\mida\Documents\XYZ logs\xyz.log I do not have any idea what id field expects, hence kept 'xyz' ```auto # ============================== Filebeat inputs =============================== filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. # filestream is an input for collecting log messages from files. - type: filestream # Unique ID among all inputs, an ID is required. id: xyz # Change to true to enable this input configuration. enabled: false # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* - C:\Users\mida\Documents\XYZ logs\xyz.log ``` ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 4, 2022, 11:57am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/2 "2022-10-04T11:57:10Z")
Hi @cadrija, your configuration looks good so far, but this line suggests the input is still disabled. Was this a copy/paste error? > [@cadrija](#): > > `enabled: false` ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 10, 2022, 4:56am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/3 "2022-10-10T04:56:20Z")
@weltenwort Thanks for your response. Sorry for my late reply. I am not aware about the field I followed some tutorial and there it was not mentioned to update/change so I left it as it was. Now I updated it to `enabled: false`. But still not getting the logs in Discover. ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 10, 2022, 11:49am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/4 "2022-10-10T11:49:30Z")
Just to be clear: it should be `enabled: true` to enable this input. Can you double-check and provide the filebeat log output if it still doesn't work? ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 10, 2022, 2:10pm UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/5 "2022-10-10T14:10:40Z")
@weltenwort I am sorry, it's my typing mistake. I updated it to `enabled: true`. The current filebeat inputs section: ```auto # ============================== Filebeat inputs =============================== filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. # filestream is an input for collecting log messages from files. - type: filestream # Unique ID among all inputs, an ID is required. id: xyz # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log #- c:\programdata\elasticsearch\logs\* - C:\Users\mida\Documents\XYZ logs\xyz.log # Exclude lines. A list of regular expressions to match. It drops the lines that are # matching any regular expression from the list. #exclude_lines: ['^DBG'] # Include lines. A list of regular expressions to match. It exports the lines that are # matching any regular expression from the list. #include_lines: ['^ERR', '^WARN'] # Exclude files. A list of regular expressions to match. Filebeat drops the files that # are matching any regular expression from the list. By default, no files are dropped. #prospector.scanner.exclude_files: ['.gz$'] # Optional additional fields. These fields can be freely picked # to add additional information to the crawled log files for filtering #fields: # level: debug # review: 1 ``` ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 10, 2022, 2:37pm UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/6 "2022-10-10T14:37:53Z")
That looks fine so far. Can you provide the log output of filebeat so we can see whether it logs any errors reading the file or connecting to Elasticsearch? ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 11, 2022, 4:53am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/7 "2022-10-11T04:53:20Z")
@weltenwort Today it's showing hits in Kibana from filebeat-\* but the logs are cloud metadata not the one from the log file that I intend to pull i.e. C:\Users\mida\Documents\XYZ\_logs\xyz.log (I added an underscore in the folder name XYZ\_logs, just to be sure that the space in the name is not creating any problem) I checked the logs folder & found only 1 line entry for the mentioned id and no errors. ```auto 2022-10-11T06:46:26.589Z INFO [input.filestream] compat/compat.go:113 Input 'filestream' starting {"id": "xyz"} ``` ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 11, 2022, 11:35am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/8 "2022-10-11T11:35:40Z")
Are you looking to ingest the log lines as they are continuously written by another process or is that a "finished" log file that you want to ingest retroactively? The background of that question is, that filebeat keeps track of its last position in the file as to not ingest lines twice. That information is stored in the `data` directory. Maybe it already considers the lines read during your experimentation? As I see from your paths you're running this on Windows, which comes with extra caveats regarding file truncation and identity. Maybe [Log rotation results in lost or duplicate events | Filebeat Reference [8.4] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html) applies here? Sorry if this feels like fishing in the dark, but I don't know enough about your infrastructure to narrow down the possible causes any other way. 😇 ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 11, 2022, 12:41pm UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/9 "2022-10-11T12:41:04Z")
@weltenwort Thanks for the response. For time being I am testing with a "finished" log file. If this is success then I will get the approval to try my hands on live log file that are continuously written by another process. ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 11, 2022, 2:30pm UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/10 "2022-10-11T14:30:40Z")
So maybe filebeat already marked your file as "read" in its registry (see [How does Filebeat keep the state of files?](https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html#_how_does_filebeat_keep_the_state_of_files)). You could stop filebeat and delete the files in the `data` directory of filebeat to clear out its state. This assumes that the filebeat instance doesn't keep track of any other log files that would be affected, though. ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 12, 2022, 5:09am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/11 "2022-10-12T05:09:07Z")
@weltenwort Thank you for this input. Let me try this out. ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 13, 2022, 6:16am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/12 "2022-10-13T06:16:14Z")
@weltenwort Tried with different log files, but still no luck 😔 ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [October 13, 2022, 12:15pm UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/13 "2022-10-13T12:15:10Z")
hm 🤔 The next thing I would do is to turn the filebeat log level up to debug and take a look at its log output to see if it reports anything that might be related. Would you be able to provide that log output after a restart of filebeat? ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [October 17, 2022, 6:26am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/14 "2022-10-17T06:26:36Z")
@weltenwort Thanks for the guidance. After Filebeat finds my log file, this is the part of logs that prints over & over. ```auto {"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:45:57.540Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:01.448Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:07.460Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:07.461Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:11.452Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:18.743Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":139},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"debug","@timestamp":"2022-10-14T12:46:18.744Z","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":209},"message":"Found 1 paths","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2022-10-14T12:46:20.269Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":185},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":1531,"time":{"ms":1531},"value":1531},"user":{"ticks":1406,"time":{"ms":1406}}},"info":{"ephemeral_id":"02ca6067-8f48-493d-8172-3f916076f7ec","name":"filebeat","uptime":{"ms":51335},"version":"8.4.3"},"memstats":{"gc_next":19892080,"memory_alloc":11033600,"memory_sys":27429896,"memory_total":58642840,"rss":58712064},"runtime":{"goroutines":32}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"handles":{"open":230}}},"ecs.version":"1.6.0"}} {"log.level":"debug","@timestamp":"2022-10-14T12:46:21.877Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/filestream.go","file.line":131},"message":"End of file reached: C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log; Backoff now.","service.name":"filebeat","id":"xyz_logs","source_file":"filestream::xyz_logs::native::196609-817-448642358","path":"C:\\Users\\adri\\Documents\\XYZ_logs\\XYZ.log","state-id":"native::196609-817-448642358","ecs.version":"1.6.0"} ``` ---
**Author:** ![weltenwort](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/weltenwort/32/53885_2.png) [@weltenwort](https://discuss.elastic.co/u/weltenwort)\ **Post date:** [November 8, 2022, 11:52am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/15 "2022-11-08T11:52:01Z")
Looks like it "knows" the file has not changed. Did you delete the cursor information (stored in the `data` directory)? After deleting the state it should re-index the file once and then do nothing until the file changes. ---
**Author:** ![cadrija](https://avatars.discourse-cdn.com/v4/letter/c/dc4da7/32.png) [@cadrija](https://discuss.elastic.co/u/cadrija)\ **Post date:** [November 10, 2022, 5:50am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/16 "2022-11-10T05:50:26Z")
@weltenwort Thank for the response. I uninstalled & deleted all traces of 7.x Filebeat then installed 8.x Filebeat but still getting same message in logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 8, 2022, 5:51am UTC](https://discuss.elastic.co/t/how-to-ingest-a-log-file-from-source-system-to-kibana-through-filebeat/315544/17 "2022-12-08T05:51:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.