Cant filter out events?

[jcor]

Hi all,
We're trying to configure winlog beats to drop info level logs but seem to be missing something. Ive found a post that is trying to do the same thing as we are but their config isnt working. So I'm guessing my logic is broken?

    winlogbeat.event_logs:
      - name: Application
        ignore_older: 72h

      - name: System

      - name: Security
        processors:
          - drop_event.when.or:
              - equals.log.level: 'information' # Filtering out information level logs
              - equals.winlog.event_id: '5145' # Filtering network share object access
              #- equals.winlog.event_id: '4656'
          - script:
              lang: javascript
              id: security
              file: ${path.home}/module/security/config/winlogbeat-security.js

      - name: Microsoft-Windows-Sysmon/Operational
        processors:
          - script:
              lang: javascript
              id: sysmon
              file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js

      - name: Windows PowerShell
        event_id: 400, 403, 600, 800
        processors:
          - script:
              lang: javascript
              id: powershell
              file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

      - name: Microsoft-Windows-PowerShell/Operational
        event_id: 4103, 4104, 4105, 4106
        processors:
          - script:
              lang: javascript
              id: powershell
              file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

      - name: ForwardedEvents
        tags: [forwarded]
        processors:
          - script:
              when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
              lang: javascript
              id: sysmon
              file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
          - script:
              when.equals.winlog.channel: Windows PowerShell
              lang: javascript
              id: powershell
              file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
          - script:
              when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
              lang: javascript
              id: powershell
              file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

[mazoutte]

Hello,

First, you need to check how is formated the name of the field for the log level in your data.
I have a doubt on log.level: 'information' , depending of the version of winlogbeat, it can be different.

Can you be more precise about what you want to achieve ?
You said you don't want all information level, but only for security Logs ? Or for every log categories?
Secondly you want to drop event id 5145 from security logs, right ?

So perhaps dropping information level, is less efficient (memory/cpu) for the agent, than configuring only the log levels you want before the drop processor.

If you want to have , for security logs everything except informational, and dropping 5146 whatever the log level :

- name: Security
  level: critical, error, warning
  ignore_older: 48h
  processors:
  - drop_event.when.or:
    - equals.winlog.event_id: 5145

I will have a look next week, with access to some logs, and help you more precisely.

Regards,
Luc

[mazoutte]

Hello,

So I checked on some logs, and the field log.level: 'information' is correct on the latest winlogbeat version.
Your config file should be working.
However you said it was not working as expected, how did you figure it ?
And what do you expect precisely ?

[jcor]

We have been able to drop events by naming specific event ids. Ideally we would like to drop larger sets of events such as info level items.

We have been able to drop events by naming events by id but unable to do the same with info level for example.

All the guides say I can but in reality nothing has worked so far.

[jcor]

Hey Mazoutte,
When we precisely expect is to not having information level logs being shipped.

[mazoutte]

Hi,

1. Regarding the config file, the drop sections will work ; but it's affecting only the logs from the security logs, not for system and app logs.
2. What's your version of Winlogbeat ?
3. did you try the suggestion I mentionned with the config file ?

[jcor]

1. The items we are trying to drop are info level in the security event log, so they should be dropped based off our logic
2. Our winlogbeat version is "winlog version 7.9.1 "
3. We have thought about dropping logs to only log what we want.

[mazoutte]

It should work, I don't see why it's not dropping the info level...

However did you try my suggestion ?

level: critical, error, warning

[jcor]

@mazoutte

  level: critical, error, warning

yeah we added that to the config when added that post and we confirmed its still sending info level

[mazoutte]

hmm, this sounds pretty strange actually.
Quick questions :
Do you restart the winlogbeat service after you modify the config file ?
Are you sure info logs are sent from the agent/machine you're trying to configure ?

[jcor]

restarting the service did not change anything

i double checked kibana and im still seeing info level from machine in question with the level: changes you suggested

[jcor]

so strange update. It works now. The weirdest part is all the changes I've made today are things we have done in the past. We've previously made the same/similar changes over a weekend and let it sit and nothing changes. But this time it worked. Must be user error on my part of a really strange bug.

[mazoutte]

Good news.
Make sure you always restart the service after any config changes.
As well the modification will only apply to new events ; "old" ones are already present in elasticsearch and changing config has no effect on these "past" events.

[jcor]

• Sometimes we've had configs not take even after restarting the service which is strange. I have no idea.

• Yeah I'm aware of that fact, normally we filter on last 5 minutes after waiting to account for old data still possibly showing up.
  Thanks for the help and working with me. Its lovely having someone help as opposed to saying hey heres a link to the documentation that you've probably already read.

[mazoutte]

Thank You for the feedback, and you're welcome.
I wish you more luck with reloading the config next time