Limiting Initial Scope of Winlogbeat

[rgibson (Rob Gibson)]

Is there a configuration option to restrict the initial ingestion of Winlogbeat to something less than the entirety of the logs, perhaps by date?

For example, I am trying to onboard about 350 Windows servers, and when I add the Winlogbeat service to a single system, I get all of the logs which are stored on that server. In my example, I have a server whose logs go back to 2012, but I have a curator job which purges logs older than 180 days.

Can I limit Winlogbeat to only look at log entries newer than 180 days, in my case?

As this stands, I have quadrupled my indices for the day until my purge happens, leading to delays as my data nodes work to keep up with replicating shards. These delays keep the cluster in a yellow or green status for dozens of minutes.

[mourlos (Harry)]

Hi,

Yes it is possible. This is part of the basic configuration of winlogbeat

The following is taken from the documentation: https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-winlogbeat-options.html

event_logs.ignore_older

If this option is specified, Winlogbeat filters events that are older than the specified amount of time. Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". This option is useful when you are beginning to monitor an event log that contains older records that you would like to ignore. This field is optional.

winlogbeat:
  event_logs:
    - name: Application
      ignore_older: 168h

Hope it helps.

[rgibson (Rob Gibson)]

I noticed that in the distributed configuration after you mentioned it. I have set the ignore_older flag on each of the logs and have confirmed that I am only ingesting new log entries and not causing 2-4 years of old logs per server to get ingested.

This will make my on-boarding process MUCH smoother.

Thanks!

Rob