Winlogbeat & Logstash - Ingest Pipelines

Elastic Stack Beats
1

Hi everyone,

I am having an issue with the ingest pipelines for powershell while using winlogbeat and the logstash output (both version 8.1.0). The powershell events seem not to be processed by the pipeline.

There is a config option to pass the pipeline to the Elasticsearch output but no such option for the logstash output. In the meantime I added this metadata in a logstash filter.
Did I miss something or is this usecase not supported?

But even when using this metadata to set the pipeline in the Elasticsearch output in logstash, there is no processing happening on these events. It seems there is a little typo in the winlogbeat routing pipeline. The pipeline checks that value of winlog.channel is equal to 'Windows Powershell' but I believe this should be 'Windows PowerShell'
Can anyone confirm this?

2

Do you have the pipeline parm in your Elasticsearch output config?

I've not done this yet with winlogbeat, but I have with filebeat. In filebeat, there is a field passed that contains the pipeline name, if that field exists, I pass it as part of the pipeline name. I don't have access to code samples now.

3

Yes, everything is working now. But I had to modify the built-in routing ingest pipeline for winlogbeat (in Elasticsearch) and manually add the pipeline metadata in logstash
My logstash pipeline looks something like this:

input {
  beats {
	port => 5044
  }
}

filter {
  if [@metadata][beat] == "winlogbeat"{
    mutate { add_field => { "[@metadata][pipeline]" => "winlogbeat-%{[agent][version]}-routing" } }
  } else if !([@metadata][pipeline]) {
    mutate { add_field => { "[@metadata][pipeline]" => "" } }
  }  
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    data_stream => "true"
    data_stream_type => "logs"
    data_stream_namespace => "testing"
    pipeline => "%{[@metadata][pipeline]}" 
    user => "username"
    password => "password"
  }  
}

I would prefer that winlogbeat could add this metadata itself, this seems not to be possible.

4

I feel Elastic is pushing us away from Logstash, particularly for beats traffic. I haven't worked on 8.x beats yet, they broke enough for us in 8.x I fear they probably changed enough with how I was filtering filebeat modules to break things.

Glad you got it working and gave an example for others that module traffic can go thru Logstash.

5

I created an issue for that. [Winlogbeat] Routing pipeline doesn't match powershell events · Issue #31287 · elastic/beats · GitHub

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.