Filebeat not processing all configured files if we run as service

Elastic Stack Beats
1

Hi all,

I have configured 4 input files in filebeat.yml.

If i start the filebeat via Command prompt then all the 4 files were processed and logs sent to Logstash.

But If i start file beat as a Service then only two files getting processed.

filebeat.inputs:

  • type: log
    enabled: true
    paths:

    • \networkA\logs\typeAserver\server1.log
    • \networkA\logs\typeAserver\server2.log
      fields:
      tags: NETWORKALogs
      fields_under_root: true
      close_inactive: 2h
      close_removed: true
      close_renamed: true

  • type: log
    enabled: true
    paths:

    • \networkB\logs\typeBserver\server1.log
    • \networkB\logs\typeBserver\server2.log
      fields:
      tags: NETWORKBLogs
      fields_under_root: true
      close_inactive: 2h
      close_removed: true
      close_renamed: true

    Above is my config.
    If I run as a serice then only NETWORKBLogs files are processed. NETWORKALogs are not processed. If I remove NETWORKBLogs from input then no files are processed.

    Below are the logs.

    If we see the logs , harverster started only for NETWORKBLogs.

2019-03-23T13:42:06.599+1100 INFO log/input.go:138 Configured paths: [\networkA\logs\typeAserver\server1.log \networkA\logs\typeAserver\server2.log]
2019-03-23T13:42:06.599+1100 INFO input/input.go:114 Starting input of type: log; ID: 18182795938752183068
2019-03-23T13:42:06.847+1100 INFO log/input.go:138 Configured paths: [\networkB\logs\typeBserver\server1.log \networkB\logs\typeBserver\server2.log]
2019-03-23T13:42:06.847+1100 INFO input/input.go:114 Starting input of type: log; ID: 11228700993579301930
2019-03-23T13:42:07.025+1100 INFO log/harvester.go:251 Harvester started for file: \networkB\logs\typeBserver\server1.log
2019-03-23T13:42:07.092+1100 INFO log/harvester.go:251 Harvester started for file: \networkB\logs\typeBserver\server2.log
2019-03-23T13:42:08.602+1100 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 2
2019-03-23T13:42:08.602+1100 INFO cfgfile/reload.go:140 Config reloader started
2019-03-23T13:42:08.603+1100 INFO cfgfile/reload.go:195 Loading of config files completed.

But if run thorough command line then i am not getting any issues. all 4 files are processed.

Anyone faced the above issue?

2

Could you please format the configuration using </>?
Is this possible that Filebeat previously read those two files when you ran it from the command line? Do they share registry files?
I think when you start Filebeat as a service, no new line showed up in those two "skipped" since the command line instace encountered it.

3

Initially, registry files were different for both.

Then I update the config to use the same registry by add the below lines in filebeat.yml file.

path.data: C:\ProgramData\filebeat

But, Both the times NETWORKALogs were not processed.

Yes. Registry files shared by both.

All the files are continuously updated with new lines by JBOSS server every 5 seconds.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.