Winlogbeat keywords filtering issue in Logstash

Hello,
I'm strugling with filtering (drop) of events with certain keywords delivered from winlogbeat (6.2.1 / 6.2.3) to logstash (6.2.1).
Events with this pattern should be droped in Logstash and I know, it would be better not to send it from winlogbeat at all...
"keywords": [
"Überwachung erfolgreich"
],

Sometimes it comes in _source
"_source": {
"tags": [
"beats_input_codec_plain_applied"
],
"keywords": [
"Überwachung erfolgreich"
],

but any filters I tried did not match, just two not working examples
if "berwachung erfolgreich" in [keywords] {
drop { }
}

if [keywords] =~ /^.berwachung erfolgreich./ {
drop { }
}

if "berwachung erfolgreich" in [_source][keywords] {
drop { }
}

This is the full json output from Kibana, slightly anonymized.
{
"_index": "winlogbeat-2018.04.09",
"_type": "doc",
"_id": "2nN4qWIBMz20DaAyNBM6",
"_version": 1,
"_score": null,
"_source": {
"tags": [
"beats_input_codec_plain_applied"
],
"thread_id": 22220,
"opcode": "Info",
"beat": {
"hostname": "Removed",
"name": "Removed",
"version": "6.2.3"
},
"type": "wineventlog",
"@version": "1",
"event_data": {
"TransmittedServices": "-",
"TargetUserSid": "Removed",
"SubjectLogonId": "0x0",
"KeyLength": "0",
"SubjectUserSid": "S-1-0-0",
"SubjectDomainName": "-",
"ProcessName": "-",
"IpPort": "65533",
"AuthenticationPackageName": "Kerberos",
"LogonType": "3",
"IpAddress": "Removed",
"TargetDomainName": "Removed",
"ImpersonationLevel": "%%1840",
"TargetLogonId": "0x6d26036a",
"LmPackageName": "-",
"ProcessId": "0x0",
"LogonGuid": "{Removed}",
"TargetUserName": "Removed",
"LogonProcessName": "Kerberos",
"SubjectUserName": "-"
},
"keywords": [
"Überwachung erfolgreich"
],
"level": "Informationen",
"process_id": 524,
"event_id": 4624,
"computer_name": "DC-REPLACED",
"provider_guid": "{Removed}",
"host": "Removed",
"source_name": "Microsoft-Windows-Security-Auditing",
"task": "Anmelden",
"record_number": "711788280",
"@timestamp": "2018-04-09T08:14:07.995Z",
"log_name": "Security",
"version": 1
},
"fields": {
"@timestamp": [
"2018-04-09T08:14:07.995Z"
]
},
"sort": [
1523261647995
]
}

------------ second version
{
"_index": "winlogbeat-2018.04.09",
"_type": "doc",
"_id": "0fOGqWIB4gPrY0VJkSbQ",
"_version": 1,
"_score": null,
"_source": {
"tags": [
"beats_input_codec_plain_applied"
],
"keywords": [
"Überwachung erfolgreich"
],
"level": "Informationen",
"opcode": "Info",
"event_id": 4624,
"process_id": 584,
"type": "wineventlog",
"beat": {
"hostname": "Removed",
"name": "Removed",
"version": "6.2.1"
},
"provider_guid": "{Removed}",
"computer_name": "Removed",
"thread_id": 5056,
"host": "Removed",
"source_name": "Microsoft-Windows-Security-Auditing",
"@version": "1",
"task": "Anmelden",
"event_data": {
"TransmittedServices": "-",
"TargetUserSid": "Removed",
"SubjectLogonId": "0x0",
"KeyLength": "0",
"SubjectUserSid": "S-1-0-0",
"SubjectDomainName": "-",
"ProcessName": "-",
"IpPort": "49216",
"AuthenticationPackageName": "Kerberos",
"LogonType": "3",
"LmPackageName": "-",
"TargetDomainName": "Removed",
"IpAddress": "Removed",
"TargetLogonId": "0xa46c7b68",
"ProcessId": "0x0",
"TargetUserName": "Removed",
"LogonGuid": "{2338B7DD-65B0-3DEC-5787-ABDCAAA843EC}",
"LogonProcessName": "Kerberos",
"SubjectUserName": "-"
},
"record_number": "744583588",
"@timestamp": "2018-04-09T08:29:48.865Z",
"log_name": "Security"
},
"fields": {
"@timestamp": [
"2018-04-09T08:29:48.865Z"
]
},
"sort": [
1523262588865
]
}

but any filters I tried did not match, just two not working examples
if "berwachung erfolgreich" in [keywords] {
drop { }
}

The keywords array doesn't contain the string "berwachung erfolgreich". Have you tried "Überwachung erfolgreich" instead?

if [keywords] =~ /^.berwachung erfolgreich./ {
drop { }
}

You can't perform regexp matches against an array.

if "berwachung erfolgreich" in [_source][keywords] {
drop { }
}

Your event doesn't have a _source field.

Hi Magnus
thank you for your answer, tried your suggestion but did not get success yet.
above are two sample kibana json outputs, one of them seemed to me as it is in _source but I might be wrong in that.

I did retry this three filters for testing:
if "Überwachung erfolgreich" in [keywords] {
drop { }
}

if "*berwachung erfolgreich" in [keywords] {
drop { }
}

if "\xC3\x9Cberwachung erfolgreich" in [keywords] {
drop { }
}

the last one is based on the output on the console, which I added here as well, none of them was successfull.

Apr 11 06:51:54 host-replaced logstash: "beat" => {
Apr 11 06:51:54 host-replaced logstash: "name" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "hostname" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "version" => "6.2.3"
Apr 11 06:51:54 host-replaced logstash: },
Apr 11 06:51:54 host-replaced logstash: "host" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "record_number" => "712930379",
Apr 11 06:51:54 host-replaced logstash: "event_data" => {
Apr 11 06:51:54 host-replaced logstash: "SubjectUserSid" => "S-1-0-0",
Apr 11 06:51:54 host-replaced logstash: "SubjectUserName" => "-",
Apr 11 06:51:54 host-replaced logstash: "SubjectDomainName" => "-",
Apr 11 06:51:54 host-replaced logstash: "IpAddress" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "TargetUserSid" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "TransmittedServices" => "-",
Apr 11 06:51:54 host-replaced logstash: "AuthenticationPackageName" => "Kerberos",
Apr 11 06:51:54 host-replaced logstash: "ProcessId" => "0x0",
Apr 11 06:51:54 host-replaced logstash: "LogonGuid" => "{Removed}",
Apr 11 06:51:54 host-replaced logstash: "LmPackageName" => "-",
Apr 11 06:51:54 host-replaced logstash: "IpPort" => "59413",
Apr 11 06:51:54 host-replaced logstash: "LogonType" => "3",
Apr 11 06:51:54 host-replaced logstash: "ImpersonationLevel" => "%%1833",
Apr 11 06:51:54 host-replaced logstash: "TargetDomainName" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "SubjectLogonId" => "0x0",
Apr 11 06:51:54 host-replaced logstash: "TargetLogonId" => "0x72a51ab4",
Apr 11 06:51:54 host-replaced logstash: "TargetUserName" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "KeyLength" => "0",
Apr 11 06:51:54 host-replaced logstash: "LogonProcessName" => "Kerberos",
Apr 11 06:51:54 host-replaced logstash: "ProcessName" => "-"
Apr 11 06:51:54 host-replaced logstash: },
Apr 11 06:51:54 host-replaced logstash: "@version" => "1",
Apr 11 06:51:54 host-replaced logstash: "computer_name" => "Removed",
Apr 11 06:51:54 host-replaced logstash: "thread_id" => 1436,
Apr 11 06:51:54 host-replaced logstash: "provider_guid" => "{Removed}",
Apr 11 06:51:54 host-replaced logstash: "event_id" => 4624,
Apr 11 06:51:54 host-replaced logstash: "log_name" => "Security",
Apr 11 06:51:54 host-replaced logstash: "opcode" => "Info",
Apr 11 06:51:54 host-replaced logstash: "source_name" => "Microsoft-Windows-Security-Auditing",
Apr 11 06:51:54 host-replaced logstash: "level" => "Informationen"
Apr 11 06:51:54 host-replaced logstash: }
Apr 11 06:51:56 host-replaced logstash: {
Apr 11 06:51:56 host-replaced logstash: "process_id" => 584,
Apr 11 06:51:56 host-replaced logstash: "computer_name" => "Removed",
Apr 11 06:51:56 host-replaced logstash: "thread_id" => 5604,
Apr 11 06:51:56 host-replaced logstash: "task" => "Anmelden",
Apr 11 06:51:56 host-replaced logstash: "keywords" => [
Apr 11 06:51:56 host-replaced logstash: [0] "\xC3\x9Cberwachung erfolgreich"
Apr 11 06:51:56 host-replaced logstash: ],
Apr 11 06:51:56 host-replaced logstash: "provider_guid" => "{Removed}",
Apr 11 06:51:56 host-replaced logstash: "@timestamp" => 2018-04-11T04:51:52.664Z,
Apr 11 06:51:56 host-replaced logstash: "type" => "wineventlog",
Apr 11 06:51:56 host-replaced logstash: "event_id" => 4624,

Works fine here.

$ cat test.config 
input { stdin { codec => json_lines } }
output { stdout { codec => rubydebug } }
filter {
  if "Überwachung erfolgreich" in [keywords] {
    drop { }
  }
}
$ cat data 
{"keywords": ["Überwachung erfolgreich"]}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

using the same setup as you and get an output.

/usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f windows.conf < data
Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
{
"host" => "Removed",
"@timestamp" => 2018-04-11T06:44:40.956Z,
"keywords" => [
[0] "\xC3\x9Cberwachung erfolgreich"
],
"@version" => "1"
}

What's the output of locale? It looks like you're not running a UTF-8-based locale.

locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

Okay, weird. Then I don't know what the problem is.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.