Event Log Record Number Tracking

Elastic Stack Beats
1

Hi,

For Winlogbeat there is a file in c:\ProgramData\Winlogbeat.winlogbeat.yml that shows you the last record sent to Elasticsearch from the machine. I noticed that if you delete this file it will not resend the events to ES and instead will recreate the file with the last record sent. I assume there is also record keeping in Elasticsearch but I can't find where it is located. If i delete the entire index and the .winlogbeat.yml the event log data will resend , but without deleting the index it will not resend.

Thanks!

2

Did you stop Winlogbeat before deleting the file? No state is stored in ES. That file holds all of the state that Winlogbeat will use to resume during a restart.

3

I did and double checked to make sure the service was in a stopped state before removing the file. When it started the file back up it was right back where it left off before deletion.

4

How are you determining where Winlogbeat is starting from?

Are you using ignore_older? Please share the config you are using.

5

I am pulling the entire security log filtering on event_id: 4624

winlogbeat.event_logs:

  • name: Application
    ignore_older: 72h
  • name: Security
    event_id: 4624
  • name: System

After some more testing i noticed a few things -

  • Stopped the Service -> Deleted the .winlogbeat.yml -> Started the Service
  • The File was recreated at the same record count before being deleted but it is re-sending the Event logs to ES which led the confusion. On another machine the record count would reset when the file was deleted and increase as the logs were being sent to ES... Not sure why in this case it goes straight to the last record even though it hasn't been sent yet.

update_time: 2017-11-03T13:21:21.6228552Z
event_logs:

  • name: Application
    record_number: 363121
    timestamp: 2017-11-03T13:20:53Z
  • name: Security
    record_number: 543894019
    timestamp: 2017-11-03T13:21:16.7638552Z
  • name: System
    record_number: 682637
    timestamp: 2017-11-03T13:20:53.2468552Z
6

It's going to be very hard to observe the starting state by looking at this file because it is updated so quickly after Winlogbeat is started. If you want to confirm the first record_number then I recommend enabling the file output and looking at the first event that gets written.

  1. Stop Winlogbeat
  2. Enable the file output in your config.
  3. Delete the registry file (aka .winlogbeat.yml).
  4. Delete any previously created output files.
  5. Start Winlogbeat.
  6. Observe the record_number of the first event written to the file output. (Note that these output files automatically rotate so the first record will be at the beginning of the oldest file.)

The record_number in the registry file is updated after the event has been confirmed to have been received by the output. So when Elasticsearch responds with a 200 OK to Winlogbeat's _bulk indexing request then Winlogbeat will eventually persist this number. This is how Winlogbeat achieves its at-least-once delivery semantics.

This is the config to use for the file output.

output.file:
  path: 'C:/ProgramData/winlogbeat/output'
  filename: events.json
  number_of_files: 50
7

Thanks Andrew! Appreciate all the help.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.