Filebeat not getting new IIS events after initial load

Elastic Stack Beats
1

I'm trying to stream IIS advanced logs via Filebeat to Logstash -> Elasticsearch. If I delete the registry files and restart filebeat it loads all of the data, but does not load any additional log lines after the initial read. I'm not using the actual IIS module since I'm sending to Logstash first for parsing purposes. My filebeat input is this:

// - type: log
// - 'c:\inetpub\logs\AdvancedLogs*.log'
// fields:
// logtype: iis
// fields_under_root: true

Can someone tell me what I'm doing wrong? I'm running Filebeat 6.4.2 and Logstash/Elasticsearch 6.4.1. However this problem presented in previous versions as well.

2

Hello @rstasiunas I think your input definition is not correct, it should contains an enabled field, I believe the following will work for you.

- type: log
  enabled: true
  paths:
    - c:\inetpub\logs\AdvancedLogs*.log
  fields:
    logtype: iis
3

Apologies, that line is included in my config. I got a little aggressive on the copy/paste. Updated config is below.

filebeat.inputs:
- type: log
  enabled: true
    - 'c:\inetpub\logs\AdvancedLogs\*.log'
  fields:
    logtype: iis
  fields_under_root: true
4

Looking at your second config fragment I do not see anything wrong.

Can you add the starting log when Filebeat is started in debug mode and is there any error in the log?

./filebeat -v -e -d "*"
5

I do not see any errors in the log output. When restarting filebeat, it gets the latest events since the last restart, but nothing new while it's running. It appears it gets to the end of the file and then just stops sending more data. The process continues to run, it just does not send any new logs since it was started.

6

@rstasiunas New events are still appended in the logs? There is not connection issues or timeout related messages in the logs?

7

That's correct. It updates the registry file with the latest changes when it's started and sends the latest information. However if new events are appended after the initial read, those appended events are not recognized until Filebeat is restarted. There are no connection or timeout messages in the debug/info log.

8

Is there any errors message on the Logstash side?

9

None that I'm seeing. The Info level logs on the filebeat side shows the following:

2018-10-25T11:07:31.481-0400    INFO    [monitoring]    log/log.go:141  Non-zero metrics in the last 30s        {"monito
ring": {"metrics": {"beat":{"cpu":{"system":{"ticks":327},"total":{"ticks":779,"value":779},"user":{"ticks":452}},"info"
:{"ephemeral_id":"4ea49c67-8a37-417c-9cef-ac681b6c3daf","uptime":{"ms":4980211}},"memstats":{"gc_next":4194304,"memory_a
lloc":1743312,"memory_total":11593120,"rss":16384}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"co
nfig":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":1}}},"registrar":{"states":{"current":26}}}}}
2018-10-25T11:08:01.486-0400    INFO    [monitoring]    log/log.go:141  Non-zero metrics in the last 30s        {"monito
ring": {"metrics": {"beat":{"cpu":{"system":{"ticks":327},"total":{"ticks":779,"value":779},"user":{"ticks":452}},"info"
:{"ephemeral_id":"4ea49c67-8a37-417c-9cef-ac681b6c3daf","uptime":{"ms":5010211}},"memstats":{"gc_next":4194304,"memory_a
lloc":1792128,"memory_total":11641936}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"modu
le":{"running":0}},"pipeline":{"clients":1,"events":{"active":1}}},"registrar":{"states":{"current":26}}}}}

It's almost like it reaches the end of the file and then never captures the appended events or re-harvests the file to look for changes.

10

If you start filebeat with -e -d "harvester" it will give you a lot more log debug. I am interested to know if FB is closing the log file in question and why it does that.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.