Winlogbeat 8.12.0 not working with filters

[kriti_dabas]

• Currently I am using winlogbeat service version 8.12.0 on my system.
• I am trying to apply the same filters on it which are working perfectly fine with winlogbeat 7.14 version.
• system - kafka topic - logstash - elasticsearch - kibana this is how i want the flow to be for my winlogbeat logs.
• But the logs are not getting parsed using these filters so basically they are not going any further from the input pipeline of logstash because of the filters.
• Following are the filter names :-

all-filter.conf
all-fingerprint-hash-filter.conf
disabled
dst-ip-filter.conf
dst-nat-ip-filter.conf
event.conf
filter.conf
final-parse-detection-and-cleanup-filter.conf
fingerprints-command_line-filter.conf
fingerprints-network_community_id-filter.conf
fingerprints-powershell-filter.conf
helk-domains-and-hostnames-enrichments_and_additions-filter.conf
meta-command_line-enrichment_and_additions-filter.conf
meta-powershell-enrichment_and_additions-filter.conf
ruby
src-ip-filter.conf
src-nat-ip-filter.conf
winevent-application-filter.conf
winevent-cleanup-catchall-guids-filter.conf
winevent-cleanup-lowercasing-windows-filter.conf
winevent-cleanup-no-dashes-only-values-filter.conf
winevent-conversions-catchall-process-ids-filter.conf
winevent-conversions-ip-conversions-basic-filter.conf
winevent-hostname-cleanups-filter.conf
winevent-logon-ids-conversions-filter.conf
winevent-main-output.conf
winevent-powershell-filter.conf
winevent-process-ids-conversions-filter.conf
winevent-process-path-split-to-name-filter.conf
winevent-rename-catchall-general-filter.conf
winevent-rename-catchall-processes-filter.conf
winevent-rename-catchall-process-guids-filter.conf
winevent-security-conversions-filter.conf
winevent-security-filter.conf
winevent-security-schtasks-filter.conf
winevent-sysmon-filter.conf
winevent-system-filter.conf
winevent-user-ids-conversions-filter.conf
winevent-username-final-modifcations-filter.conf
winevent-winlogbeats-filter.conf
winevent-wmiactivity-filter.conf
winevent-wmiactivity-output.conf
winlogbeat-input.conf

[strawgate (William Easton)]

I would recommend setting up a test winglogbeat with a test logstash instance and enable debug logging so you can see why the messages aren't the format that logstash is expecting.

Nobody here is going to be able to offer much help without a lot more information about what is happening.

That being said, I believe in 7.x winlogbeat, processing of the security events was done on the client in a JavaScript processor and in 8.x it's done in ingest pipelines in elasticsearch. This means some of the fields you might be relying on are no longer in the event when it hits logstash.

In the latest version of logstash there is the ability to run elasticsearch Ingest pipelines within logstash which might unblock your usecase. See more: Using Logstash with Elastic Integrations (Beta) | Logstash Reference [8.12] | Elastic

[leandrojmp (Leandro Pereira)]

@kriti_dabas

system - kafka topic - logstash - elasticsearch - kibana this is how i want the flow to be for my winlogbeat logs.

What is the Logstash version you are using? There are breaking changes in both Winlogbeat 8.X and Logstash 8.X that can impact your processing.

As @strawgate mentioned one of the breaking changes is that on Winlogbeat 8.X the processing was moved from the edge (winlogbeat) to the ingest node (elasticsearch using ingest pipelines) as you can check it here, with links for the PR that changed that.

So the output data from Winlogbeat 7.X will be different from Winlogbeat 8.X.

[kriti_dabas]

Thankyou for the help.
My filters are working now, I copied them from running production but
I didn't apply them in sequential order and I had changed the name of all files because of which the filters were not working.