Events being not published to logstash when using drop_event.when.not.or processor

Hi

I'm having a problem with winlogbeat not publishing events to logstash when I configure the processors for Security events so that I can specify more than the 22 limit:

  - name: Security
    ignore_older: 72h
    processors:
    - drop_event.when.not.or:
      - equals.event_id: 1100
      - equals.event_id: 1102
      - equals.event_id: 1104
      - equals.event_id: 4608-4609
      - equals.event_id: 4616
      - equals.event_id: 4624-4627
      - equals.event_id: 4634
      - equals.event_id: 4647-4649
      - equals.event_id: 4672
      - equals.event_id: 4688
      - equals.event_id: 4697
      - equals.event_id: 4703-4705
      - equals.event_id: 4720-4767
      - equals.event_id: 4778-4779
      - equals.event_id: 4783-4792
      - equals.event_id: 4886-4888
      - equals.event_id: 5140
      - equals.event_id: 5142-5144
      - equals.event_id: 5148-5149
      - equals.event_id: 5632-5633
      - equals.event_id: 6416
      - equals.event_id: 6420-6424

If I take the processors out everything makes it to logstash.

  - name: Security
    ignore_older: 72h

Just in case I had too many filters I tried just the one and it doesn't work either

  - name: Security
    ignore_older: 72h
    processors:
    - drop_event.when.not.or:
      - equals.event_id: 4624

The debug logs indicate the events are being filtered out when I have the processors configured:

2019-05-20T15:33:12.284+0100	DEBUG	[publisher]	pipeline/client.go:200	Pipeline client receives callback 'onFilteredOut' for event: %+v{2019-05-20 14:33:11.6114434 +0000 UTC null {"event":{"action":"Logon","code":4624,"created":"2019-05-20T14:33:12.284Z","kind":"event"},"log":{"level":"information"},"message":"An account was successfully logged on.\n\nSubject:\n\tSecurity ID:\t\tS-1-0-0\n\tAccount Name:\t\t-\n\tAccount Domain:\t\t-\n\tLogon ID:\t\t0x0\n\nLogon Information:\n\tLogon Type:\t\t3\n\tRestricted Admin Mode:\t-\n\tVirtual Account:\t\tNo\n\tElevated Token:\t\tYes\n\nImpersonation Level:\t\tImpersonation\n\nNew Logon:\n\tSecurity ID:\t\tS-1-5-21-664342396-1520592568-226259266-113888\n\tAccount Name:\t\t<REMOVED>\n\tAccount Domain:\t\t <REMOVED> \n\tLogon ID:\t\t0xCB4B844\n\tLinked Logon ID:\t\t0x0\n\tNetwork Account Name:\t-\n\tNetwork Account Domain:\t-\n\tLogon GUID:\t\t{<REMOVED>}\n\nProcess Information:\n\tProcess ID:\t\t0x0\n\tProcess Name:\t\t-\n\nNetwork Information:\n\tWorkstation Name:\t-\n\tSource Network Address:\t<REMOVED>\n\tSource Port:\t\t0\n\nDetailed Authentication Information:\n\tLogon Process:\t\tKerberos\n\tAuthentication Package:\tKerberos\n\tTransited Services:\t-\n\tPackage Name (NTLM only):\t-\n\tKey Length:\t\t0\n\nThis event is generated when a logon session is created. It is generated on the computer that was accessed.\n\nThe subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.\n\nThe logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network).\n\nThe New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on.\n\nThe network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.\n\nThe impersonation level field indicates the extent to which a process in the logon session can impersonate.\n\nThe authentication information fields provide detailed information about this specific logon request.\n\t- Logon GUID is a unique identifier that can be used to correlate this event with a KDC event.\n\t- Transited services indicate which intermediate services have participated in this logon request.\n\t- Package name indicates which sub-protocol was used among the NTLM protocols.\n\t- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","winlog":{"activity_id":"{0C83FCBA-5742-0002-648E-E02FDA0ED501}","api":"wineventlog","channel":"Security","computer_name":"<REMOVED>","event_data":{"AuthenticationPackageName":"Kerberos","ElevatedToken":"%%1842","ImpersonationLevel":"%%1833","IpAddress":"<REMOVED>","IpPort":"0","KeyLength":"0","LmPackageName":"-","LogonGuid":"{<REMOVED>}","LogonProcessName":"Kerberos","LogonType":"3","ProcessId":"0x0","ProcessName":"-","RestrictedAdminMode":"-","SubjectDomainName":"-","SubjectLogonId":"0x0","SubjectUserName":"-","SubjectUserSid":"S-1-0-0","TargetDomainName":"<REMOVED>","TargetLinkedLogonId":"0x0","TargetLogonId":"0xcb4b844","TargetOutboundDomainName":"-","TargetOutboundUserName":"-","TargetUserName":"<REMOVED>","TargetUserSid":"<REMOVED>","TransmittedServices":"-","VirtualAccount":"%%1843","WorkstationName":"-"},"event_id":4624,"keywords":["Audit Success"],"opcode":"Info","process":{"pid":764,"thread":{"id":860}},"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","provider_name":"Microsoft-Windows-Security-Auditing","record_id":14946,"task":"Logon","version":2}} {Security 14946 2019-05-20 14:33:11.6114434 +0000 UTC <BookmarkList>

I'm using winlogbeat 7.0.1

Any ideas?

Thanks, Kevin

Hi @nhscyberguy and welcome :slight_smile:

The event_id field is in winlog.event_id, so in principle the whole path should be used, and the rules should be then like this:

  - name: Security
    ignore_older: 72h
    processors:
    - drop_event.when.not.or:
      - equals.winlog.event_id: 4624

Hi @jsoriano and thanks

Unfortunately that didn't work. Also, the winlogbeat documentation says use "equals.event_id:" https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-winlogbeat-options.html#_literal_event_logs_event_id_literal

Just in case it helps below is the full winlogbeats.yml file (I've stripped out all the commented sections):

winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
    provider:
      - Application Error
      - Application Hang
      - EMET
  - name: System
    event_id: 1, 12, 13, 104, 1056, 6005, 6006, 7045, 7030
    ignore_older: 72h
  - name: Microsoft-Windows-AppLocker/Packaged app-Deployment
    ignore_older: 72h
  - name: Microsoft-Windows-AppLocker/Packaged app-Execution
    ignore_older: 72h
  - name: Microsoft-Windows-SmartCard-Audit/Authentication
    ignore_older: 72h
  - name: Microsoft-Windows-SMBClient/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-Sysmon/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-TaskScheduler/Operational
    provider:
      - Microsoft-Windows-TaskScheduler
    event_id: 106, 141, 142
    ignore_older: 72h
  - name: Microsoft-Windows-TerminalServices-RDPClient/Operational
    event_id: 1024
    ignore_older: 72h
  - name: Microsoft-Windows-Windows Defender/Operational
    event_id: 1006-1009, 1116-1119
    ignore_older: 72h
  - name: Microsoft-Windows-SMBClient/Operational
    ignore_older: 72h
  - name: Microsoft-Windows-PowerShell/Operational
    ignore_older: 72h
  - name: Windows Powershell
    event_id: 400, 600, 800
    ignore_older: 72h
  - name: Microsoft-Windows-WMI-Activity/Operational
    event_id: 5857-5861
    ignore_older: 72h
  - name: Microsoft-Windows-TaskScheduler/Operational
    ignore_older: 72h
  - name: Security
    ignore_older: 72h
    processors:
      - drop_event.when.not.or:
        - equals.winlog.event_id: 1100
        - equals.winlog.event_id: 1102
        - equals.winlog.event_id: 1104
        - equals.winlog.event_id: 4608
        - equals.winlog.event_id: 4609
        - equals.winlog.event_id: 4616
        - equals.winlog.event_id: 4624-4627
        - equals.winlog.event_id: 4634
        - equals.winlog.event_id: 4647-4649
        - equals.winlog.event_id: 4672
        - equals.winlog.event_id: 4688
        - equals.winlog.event_id: 4697
        - equals.winlog.event_id: 4703-4705
        - equals.winlog.event_id: 4720-4767
        - equals.winlog.event_id: 4778-4779
        - equals.winlog.event_id: 4783-4792
        - equals.winlog.event_id: 4886-4888
        - equals.winlog.event_id: 5140
        - equals.winlog.event_id: 5142-5144
        - equals.winlog.event_id: 5148-5149
        - equals.winlog.event_id: 5632-5633
        - equals.winlog.event_id: 6416
        - equals.winlog.event_id: 6420-6424

setup.template.settings:
  index.number_of_shards: 3

output.logstash:
  hosts: ["<REDACTED>:5044"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: debug

it turns out it did work ...sort of. Some of the security events included in the filter were sent but not all.

I've been testing with event id 4624 (easy to generate). I noticed that after I replied last time that event id 4634 was making it to logstash. The difference between it and 4624 is that 4634 was explicitly listed (equals.winlog.event_id: 4634) whereas 4624 is part of a range (equals.winlog.event_id: 4624-4627).

I've change the filter so that 4624 is explicitly listed and now the events are being sent to logstash:

The debug logs still show events being filtered that shouldn't be but they are defined in ranges:

`2019-05-21T10:45:45.898+0100	DEBUG	[publisher]	pipeline/client.go:200	Pipeline client receives callback 'onFilteredOut' for event: %+v{2019-05-21 09:45:44.2060769 +0000 UTC null {"event":{"action":"Logon","code":4648,"created":"2019-05-21T09:45:45.898Z","kind":"event"},"log":{"level":"information"},"message":"A logon was attempted using explicit credentials.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\t<REDACTED>$\n\tAccount Domain:\t\<REDACTED>\n\tLogon ID:\t\t0x3E7\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\n\nAccount Whose Credentials Were Used:\n\tAccount Name:\t\tDWM-3\n\tAccount Domain:\t\tWindow Manager\n\tLogon GUID:\t\t{00000000-0000-0000-0000-000000000000}\n\nTarget Server:\n\tTarget Server Name:\tlocalhost\n\tAdditional Information:\tlocalhost\n\nProcess Information:\n\tProcess ID:\t\t0x13b0\n\tProcess Name:\t\tC:\\Windows\\System32\\winlogon.exe\n\nNetwork Information:\n\tNetwork Address:\t-\n\tPort:\t\t\t-\n\nThis event is generated when a process attempts to log on an account by explicitly specifying that account’s credentials.  This most commonly occurs in batch-type configurations such as scheduled tasks, or when using the RUNAS command.","winlog":{"activity_id":"{0C83FCBA-5742-0003-CA2C-06F5B20FD501}","api":"wineventlog","channel":"Security","computer_name":"<REDACTED>","event_data":{"IpAddress":"-","IpPort":"-","LogonGuid":"{00000000-0000-0000-0000-000000000000}","ProcessId":"0x13b0","ProcessName":"C:\\Windows\\System32\\winlogon.exe","SubjectDomainName":"<REDACTED>","SubjectLogonId":"0x3e7","SubjectUserName":"<REDACTED>","SubjectUserSid":"<REDACTED>","TargetDomainName":"Window Manager","TargetInfo":"localhost","TargetLogonGuid":"{00000000-0000-0000-0000-000000000000}","TargetServerName":"localhost","TargetUserName":"DWM-3"},"event_id":4648,"keywords":["Audit Success"],"opcode":"Info","process":{"pid":692,"thread":{"id":1992}},"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","provider_name":"Microsoft-Windows-Security-Auditing","record_id":15287,"task":"Logon"}} {Security 15287 2019-05-21 09:45:44.2060769 +0000 UTC <BookmarkList>`

Are ranges supported in processor based filters?

I did test separating the ranges into individual IDs yesterday but without including ".winlog." in the processor and it didn't work.

There is a range condition to check ranges. For one of your cases it'd be something like:

      - range.winlog.event_id: { gte: 4608, lte: 4609 }

Ok, thanks. I'll hopefully get tested later today.

The documentation suggests that ranges can be defined the way I had them configured. Would be worthwhile getting this updated to clarify that and the need to use .winlog.

Format is not the same for event_id and for range conditions.

Ranges in the winlogbeat event_id option can be defined in the 4608-4609 form for convenience, but in conditions there are more options and then ranges need to be more explicit.

But yes, it could be good to add an example with ranges in the drop_event processor to avoid confusion. Would you like to open a PR or provide a working example when you have it working? This would be the file to change: https://github.com/elastic/beats/blob/master/winlogbeat/docs/winlogbeat-options.asciidoc

Will do

Thanks for the help Jaime.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.