Filtering not working

ManuelF · May 26, 2020, 6:47pm

Hi,

Running ELK 6.8.9 on premises
Output in Winlogbeat config file is Eslasticsearch (not Logstash)

I am trying to add some processing on the client side to filter only the Win events I need before sending the data to ELK. I am trying a drop_event filter following steps found in the following URL:

https://www.elastic.co/guide/en/beats/winlogbeat/current/configuration-winlogbeat-options.html

When applying the filter, no logs are received by ELK.

My goal is to send to ELK only the events listed in the config below.

winlogbeat.event_logs:

name: Application
ignore_older: 72h
name: Security
processors:
- drop_event.when.not.or:
  - equals.winlog.event_id: 4624
  - equals.winlog.event_id: 4625
  - equals.winlog.event_id: 4634
  - equals.winlog.event_id: 4648
  - equals.winlog.event_id: 4719
  - equals.winlog.event_id: 4964
  - equals.winlog.event_id: 1102
  - equals.winlog.event_id: 4720
  - equals.winlog.event_id: 4722
  - equals.winlog.event_id: 4723
  - equals.winlog.event_id: 4725
  - equals.winlog.event_id: 4728
  - equals.winlog.event_id: 4732
  - equals.winlog.event_id: 4756
  - equals.winlog.event_id: 4738
  - equals.winlog.event_id: 4740
  - equals.winlog.event_id: 4767
  - equals.winlog.event_id: 4735
  - equals.winlog.event_id: 4737
  - equals.winlog.event_id: 4755
  - equals.winlog.event_id: 4772
  - equals.winlog.event_id: 4777
  - equals.winlog.event_id: 4782
  - equals.winlog.event_id: 4616
  - equals.winlog.event_id: 4657
  - equals.winlog.event_id: 4697
  - equals.winlog.event_id: 4698
  - equals.winlog.event_id: 4699
  - equals.winlog.event_id: 4700
  - equals.winlog.event_id: 4701
  - equals.winlog.event_id: 4702
  - equals.winlog.event_id: 4946
  - equals.winlog.event_id: 4947
  - equals.winlog.event_id: 4950
  - equals.winlog.event_id: 4954
  - equals.winlog.event_id: 5025
  - equals.winlog.event_id: 5031
  - equals.winlog.event_id: 5152
  - equals.winlog.event_id: 5153
  - equals.winlog.event_id: 5155
  - equals.winlog.event_id: 5157
  - equals.winlog.event_id: 5447
name: System
name: Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurity,Microsoft-Windows-Windows Firewall With Advanced Security/ConnectionSecurityVerbose,Microsoft-Windows-Windows Firewall With Advanced Security/Firewall,Microsoft-Windows-Windows Firewall With Advanced Security/FirewallVerbose,Microsoft-Windows-Windows Firewall With Advanced Security/Network Isolation Operational

What could be wrong here?

Thank you

ManuelF · May 27, 2020, 8:39pm

I have been trying different combinations found around the forum. This time I reduced the amount of processors "drop_event", but still it is not filtering. Also considering event ID as string instead of integer. ELK keep receiving events not included on the list and also events were SubjetUserName or TargetUserName involves accounts ending in $ (service managed accounts).

If somebody could tell me what I am doing wrong....

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_event:
      when.or:
        # This filters logons from managed service accounts.
        # The trailing dollar sign is reserved for managed service accounts.
        - regexp.winlog.event_data.TargetUserName: '.*\$'
        - regexp.winlog.event_data.SubjectUserName: '.*\$'

        # This filters logon type 0 which is used for system accounts.
        - equals.winlog.event_data.LogonType: '0'

        # This filters logon type 5 which is used for service accounts.
        - equals.winlog.event_data.LogonType: '5'

        # This filters anonymous logons which are typically benign.
        # Anonymous users have extremely limited privileges.
        - equals.winlog.event_data.TargetUserName: 'ANONYMOUS LOGON'
      
      when.not.or:
        - equals.winlog.event_id: '4624'
        - equals.winlog.event_id: '4625'
        - equals.winlog.event_id: '4634'
        - equals.winlog.event_id: '4648'
        - equals.winlog.event_id: '4719'
        - equals.winlog.event_id: '4964'
        - equals.winlog.event_id: '1102'
        - equals.winlog.event_id: '4720'
        - equals.winlog.event_id: '4722'
        - equals.winlog.event_id: '4723'
        - equals.winlog.event_id: '4725'
        - equals.winlog.event_id: '4728'
        - equals.winlog.event_id: '4732'
        - equals.winlog.event_id: '4756'
        - equals.winlog.event_id: '4738'
        - equals.winlog.event_id: '4740'
        - equals.winlog.event_id: '4767'
        - equals.winlog.event_id: '4735'
        - equals.winlog.event_id: '4737'
        - equals.winlog.event_id: '4755'
        - equals.winlog.event_id: '4772'
        - equals.winlog.event_id: '4777'
        - equals.winlog.event_id: '4782'
        - equals.winlog.event_id: '4616'
        - equals.winlog.event_id: '4657'
        - equals.winlog.event_id: '4697'
        - equals.winlog.event_id: '4698'
        - equals.winlog.event_id: '4699'
        - equals.winlog.event_id: '4700'
        - equals.winlog.event_id: '4701'
        - equals.winlog.event_id: '4702'
        - equals.winlog.event_id: '4946'
        - equals.winlog.event_id: '4947'
        - equals.winlog.event_id: '4950'
        - equals.winlog.event_id: '4954'
        - equals.winlog.event_id: '5025'
        - equals.winlog.event_id: '5031'
        - equals.winlog.event_id: '5152'
        - equals.winlog.event_id: '5153'
        - equals.winlog.event_id: '5155'
        - equals.winlog.event_id: '5157'
        - equals.winlog.event_id: '5447'

MarianaD · May 28, 2020, 1:19pm

hi @ManuelF , can you try removing the winlog prefix from the field name and let us know if it worked?
ex:

- regexp.event_data.TargetUserName: '.*\$'

instead of

- regexp.winlog.event_data.TargetUserName: '.*\$'

same for the event_id , also values can be integer.

ManuelF · May 28, 2020, 6:44pm

Hi @MarianaD, I tried what you said. Removed all "winlog" strings. Restarted Winlogbeat service with no errors. ELK keeps receiving many events not listed in my filter (Ex: 5158, 5156, 4800, etc...).

With the processor working as it should, those Windows events should not reach ELK. Right?

I am available and ready to try any other solution.

Thank you

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_event:
      when.or:
        # This filters logons from managed service accounts.
        # The trailing dollar sign is reserved for managed service accounts.
        - regexp.event_data.TargetUserName: '.*\$'
        - regexp.event_data.SubjectUserName: '.*\$'

        # This filters logon type 0 which is used for system accounts.
        - equals.event_data.LogonType: '0'

        # This filters logon type 5 which is used for service accounts.
        - equals.event_data.LogonType: '5'

        # This filters anonymous logons which are typically benign.
        # Anonymous users have extremely limited privileges.
        - equals.event_data.TargetUserName: 'ANONYMOUS LOGON'
      
      when.not.or:
        - equals.event_id: 4624
        - equals.event_id: 4625
        - equals.event_id: 4634
        - equals.event_id: 4648
        - equals.event_id: 4719
        - equals.event_id: 4964
        - equals.event_id: 1102
        - equals.event_id: 4720
        - equals.event_id: 4722
        - equals.event_id: 4723
        - equals.event_id: 4725
        - equals.event_id: 4728
        - equals.event_id: 4732
        - equals.event_id: 4756
        - equals.event_id: 4738
        - equals.event_id: 4740
        - equals.event_id: 4767
        - equals.event_id: 4735
        - equals.event_id: 4737
        - equals.event_id: 4755
        - equals.event_id: 4772
        - equals.event_id: 4777
        - equals.event_id: 4782
        - equals.event_id: 4616
        - equals.event_id: 4657
        - equals.event_id: 4697
        - equals.event_id: 4698
        - equals.event_id: 4699
        - equals.event_id: 4700
        - equals.event_id: 4701
        - equals.event_id: 4702
        - equals.event_id: 4946
        - equals.event_id: 4947
        - equals.event_id: 4950
        - equals.event_id: 4954
        - equals.event_id: 5025
        - equals.event_id: 5031
        - equals.event_id: 5152
        - equals.event_id: 5153
        - equals.event_id: 5155
        - equals.event_id: 5157
        - equals.event_id: 5447

ManuelF · May 28, 2020, 8:34pm

Also, when processors are enabled, ELK is getting many (not sure if all of them) events duplicated.

ManuelF · June 5, 2020, 1:11pm

Hi,

Any other idea to resolve this. Please help.

Thank you

ManuelF · June 10, 2020, 1:24pm

@MarianaD or somebody else from Elastic Team (or a regular user) that could please help me to resolve the issue I am experiencing with Winlogbeat processors?

Thank you

mazoutte · June 11, 2020, 8:36am

Hello,

I'm not sure about using two "when" in the same processor (drop_event) - I only tried with only 1 "when" per processor.

You are in 6.8, i'm assuming your beat versions is the same. As @MarianaD said, you should not use the 'winlog' subfield in your conf.

Maybe you can try this :

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_event.when.or:
      # This filters logons from managed service accounts.
      # The trailing dollar sign is reserved for managed service accounts.
      - regexp.event_data.TargetUserName: '.*\$'
      - regexp.event_data.SubjectUserName: '.*\$'

      # This filters logon type 0 which is used for system accounts.
      - equals.event_data.LogonType: '0'

      # This filters logon type 5 which is used for service accounts.
      - equals.event_data.LogonType: '5'

      # This filters anonymous logons which are typically benign.
      # Anonymous users have extremely limited privileges.
      - equals.event_data.TargetUserName: 'ANONYMOUS LOGON'
      
      - not.or:
        - equals.event_id: 4624
        - equals.event_id: 4625
        - equals.event_id: 4634
        - equals.event_id: 4648
        - equals.event_id: 4719
        - equals.event_id: 4964
        - equals.event_id: 1102
        - equals.event_id: 4720
        - equals.event_id: 4722
        - equals.event_id: 4723
        - equals.event_id: 4725
        - equals.event_id: 4728
        - equals.event_id: 4732
        - equals.event_id: 4756
        - equals.event_id: 4738
        - equals.event_id: 4740
        - equals.event_id: 4767
        - equals.event_id: 4735
        - equals.event_id: 4737
        - equals.event_id: 4755
        - equals.event_id: 4772
        - equals.event_id: 4777
        - equals.event_id: 4782
        - equals.event_id: 4616
        - equals.event_id: 4657
        - equals.event_id: 4697
        - equals.event_id: 4698
        - equals.event_id: 4699
        - equals.event_id: 4700
        - equals.event_id: 4701
        - equals.event_id: 4702
        - equals.event_id: 4946
        - equals.event_id: 4947
        - equals.event_id: 4950
        - equals.event_id: 4954
        - equals.event_id: 5025
        - equals.event_id: 5031
        - equals.event_id: 5152
        - equals.event_id: 5153
        - equals.event_id: 5155
        - equals.event_id: 5157
        - equals.event_id: 5447

system · July 9, 2020, 8:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.