Winlogbeat modules with logstash output

Elastic Stack Beats
1

Hello,

I'm currently moving from Graylog solution to ELK stack solution for the security logs centralization.

On Graylog, I was using Winlogbeat 7.12.1 and now I'm using Winlogbeat 8.13.2.

I've successfully translated the 7.X winlogbeat configuration file to 8.X configuration especially for the security module BUT I'm stuck at some point or maybe it is a normal behaviour.

  • Old Winlogbeat 7.X config:
- name: Security
     event_id: 4608, 4610, 4611, 4614, 4622, 4697, 4719, 4817, 4826, 4902, 4904, 4905, 4906, 4908, 4912
     ignore_older: 48h
     tags: [windowsos]
     processors:
      - script:
          lang: javascript
          id: security
          file: C:\Program Files\Graylog\sidecar\module\security\config\winlogbeat-security.js
      - drop_event.when:
           and:
             - equals.winlog.event_data.SubjectUserSid: 'S-1-5-18'

output.logstash:
  hosts: ["graylog.lab.lan:5044"]
  • New config
- name: Security
     id: security-windows-os
     event_id: 4608, 4610, 4611, 4614, 4622, 4697, 4719, 4817, 4826, 4902, 4904, 4905, 4906, 4908, 4912
     ignore_older: 48h
     tags: [windowsos]
     processors:
      - drop_event.when:
           and:
             - equals.winlog.event_data.SubjectUserSid: 'S-1-5-18'

output.logstash:
  hosts: ["elk.lab.lan:5044"]

As the call of winlogbeat modules changed since 8.x, is there a way to says to winlogbeat agent to use the security and powershell modules ?

If I chose the output to be elastic, I can decide to use pipelines and security modules with the pipeline setting.

pipeline: "winlogbeat-%{[agent.version]}-routing"

But I want to send events to Logstash first, then writing GROK pattern to parse/add additionnal missing field OR to create fields based on certains conditions.

The documentation says

All module processing is handled via Elasticsearch Ingest Node pipelines
If Winlogbeat is sending events to Logstash or another output you need to load the ingest pipelines with the setup command or manually.

When I run the winlogbeat setup --pipelines, I got this error below because it tries to setup through elasticsearch protocol not logstash.

Invalid version of beats protocol: 69

When the output is logstash, how I can verify that winlogbeat is using the modules ?

Thank you

2

Howdy @s0p4L1n3 ,

Have you seen this documentation page: Working with Winlogbeat Modules | Logstash Reference [8.13] | Elastic ?

Specifically the note after it shows the winlogbeat setup command:

A connection to Elasticsearch is required for this setup step because Winlogbeat needs to load the ingest pipelines into Elasticsearch. If necessary, you can temporarily disable your configured output and enable the Elasticsearch output before running the command.

3

Yes I read it. I did at first set up Elastic output so I assume it already ingested the pipelines.

The documentation also says:

Winlogbeat’s default config file contains the option to send all events to the routing pipeline. If you remove this option then the module processing will not be applied.
output.elasticsearch.pipeline: winlogbeat-%{[agent.version]}-routing

There is no winlogbeat-%{[agent.version]}-routing for Logstash, does winlogbeat process the modules when logstash is used as output ?

4

And If I change my logstash conf from:

input {
        beats {
                port => 5044
        }
}

## Add your filters / logstash plugins configuration here

output {
    elasticsearch {
      hosts => "elasticsearch:9200"
      user => "logstash_internal"
      password => "${LOGSTASH_INTERNAL_PASSWORD}"
    }
  }

to:

input {
        beats {
                port => 5044
        }
}

## Add your filters / logstash plugins configuration here

output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => "elasticsearch:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
      pipeline => "%{[@metadata][pipeline]}"
      user => "logstash_internal"
      password => "${LOGSTASH_INTERNAL_PASSWORD}"
    }
  } else {
    elasticsearch {
      hosts => "elasticsearch:9200"
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}"
      action => "create"
      user => "logstash_internal"
      password = "${LOGSTASH_INTERNAL_PASSWORD}"
    }
  }
}

I got some errors:

 Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 36, column 16 (byte 758) after output {\n  if [@metadata][pipeline] {\n    elasticsearch {\n      hosts => \"elasticsearch:9200\"\n      manage_template => false\n      index => \"%{[@metadata][beat]}-%{[@metadata][version]}\" \n      action => \"create\" \n      pipeline => \"%{[@metadata][pipeline]}\" \n      user => \"logstash_internal\"\n      password => \"${LOGSTASH_INTERNAL_PASSWORD}\"\n    }\n  } else {\n    elasticsearch {\n      hosts => \"elasticsearch:9200\"\n      manage_template => false\n      index => \"%{[@metadata][beat]}-%{[@metadata][version]}\" \n      action => \"create\"\n      user => \"logstash_internal\"\n      password ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:931:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
5

A character was missing for my logstash conf, but now I have another error:

Retrying failed action {:status=>403, :action=>["create", {:_id=>nil, :_index=>"winlogbeat-8.13.2", :routing=>nil}, {"tags"=>["filesystem", "beats_input_codec_plain_applied"], "agent"=>{"id"=>"7d7b46ea-dbef-4ea3-ac8c-3e1ca717e983", "ephemeral_id"=>"d3a47579-53fc-4c13-ad98-3522aada7d3e", "version"=>"8.13.2",

It look like it try to send the routing module but fails

logstash       | [2024-04-30T11:03:52,203][INFO ][logstash.outputs.elasticsearch][main][922faaeb295f383636b4b7c407971a1af4b56664c571c61ae2fc4d513c4486c8] Retrying individual bulk actions that failed or were rejected by the previous bulk request {:count=>125}

I have the reason:

reason"=>"action [indices:data/write/bulk[s]] is unauthorized for user [logstash_internal] with effective roles [logstash_writer] on indices [winlogbeat-8.13.2], this action is granted by the index privileges [create_doc,create,delete,index,write,all]"}}

I will try to fix the permissions and post the outcome.

6

I fixed my problem that was simple in the end, I just didn't read my logstash logs till the end of the line where the problem was displayed.

I've added the index privileges to the logstash_writer role which is used by my logstash user.

image
image3353×184 12.3 KB