Processing of multiple once created files

Elastic Stack Beats
1

hello,
my application creates the log records as separate files, each log record is in one file. I tried to configure FileBeat to process them and send to logstash, but with no success.
I used the latest available version of FileBeat (6.3.1) on Windows.
My configuration look like:

filebeat.inputs:
- type: log
  paths:
    - C:\Temp\Errors\*.json
  json:
    keys_under_root: true
    overwrite_keys: true
  fields_under_root: true
  fields:
    "@by": filebeat
    "@env": dev-vagrant
  close_eof: true

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  enabled: true
  hosts: ["localhost:10300"]
  ssl:
    enabled: true
    verification_mode: none

Thank you for your response.

2

Hi @pazderak and welcome :slight_smile:

What problems are you having? No log is being sent?

You can check if filebeat is reading the logs by reading filebeat logs, there should be a line about harvesting started for each line.

Something that could be happening if files are being harvested but not sent is that filebeat needs files to be ended by a newline, could it be that your log files don't have it?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.