# Winlogbeat keywords filtering issue in Logstash **URL:** \ **Category:** Logstash\ **Created:** [April 9, 2018, 8:37am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278 "2018-04-09T08:37:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sru](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sru](https://discuss.elastic.co/u/sru)\ **Post date:** [April 9, 2018, 8:37am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/1 "2018-04-09T08:37:28Z")
Hello, I'm strugling with filtering (drop) of events with certain keywords delivered from winlogbeat (6.2.1 / 6.2.3) to logstash (6.2.1). Events with this pattern should be droped in Logstash and I know, it would be better not to send it from winlogbeat at all... "keywords": [ "Überwachung erfolgreich" ], Sometimes it comes in \_source "\_source": { "tags": [ "beats\_input\_codec\_plain\_applied" ], "keywords": [ "Überwachung erfolgreich" ], but any filters I tried did not match, just two not working examples if "berwachung erfolgreich" in [keywords] { drop { } } if [keywords] =~ /^._berwachung erfolgreich._/ { drop { } } if "berwachung erfolgreich" in [\_source][keywords] { drop { } } This is the full json output from Kibana, slightly anonymized. { "\_index": "winlogbeat-2018.04.09", "\_type": "doc", "\_id": "2nN4qWIBMz20DaAyNBM6", "\_version": 1, "\_score": null, "\_source": { "tags": [ "beats\_input\_codec\_plain\_applied" ], "thread\_id": 22220, "opcode": "Info", "beat": { "hostname": "Removed", "name": "Removed", "version": "6.2.3" }, "type": "wineventlog", "@version": "1", "event\_data": { "TransmittedServices": "-", "TargetUserSid": "Removed", "SubjectLogonId": "0x0", "KeyLength": "0", "SubjectUserSid": "S-1-0-0", "SubjectDomainName": "-", "ProcessName": "-", "IpPort": "65533", "AuthenticationPackageName": "Kerberos", "LogonType": "3", "IpAddress": "Removed", "TargetDomainName": "Removed", "ImpersonationLevel": "%%1840", "TargetLogonId": "0x6d26036a", "LmPackageName": "-", "ProcessId": "0x0", "LogonGuid": "{Removed}", "TargetUserName": "Removed", "LogonProcessName": "Kerberos", "SubjectUserName": "-" }, "keywords": [ "Überwachung erfolgreich" ], "level": "Informationen", "process\_id": 524, "event\_id": 4624, "computer\_name": "DC-REPLACED", "provider\_guid": "{Removed}", "host": "Removed", "source\_name": "Microsoft-Windows-Security-Auditing", "task": "Anmelden", "record\_number": "711788280", "@timestamp": "2018-04-09T08:14:07.995Z", "log\_name": "Security", "version": 1 }, "fields": { "@timestamp": [ "2018-04-09T08:14:07.995Z" ] }, "sort": [ 1523261647995 ] } ------------ second version { "\_index": "winlogbeat-2018.04.09", "\_type": "doc", "\_id": "0fOGqWIB4gPrY0VJkSbQ", "\_version": 1, "\_score": null, "\_source": { "tags": [ "beats\_input\_codec\_plain\_applied" ], "keywords": [ "Überwachung erfolgreich" ], "level": "Informationen", "opcode": "Info", "event\_id": 4624, "process\_id": 584, "type": "wineventlog", "beat": { "hostname": "Removed", "name": "Removed", "version": "6.2.1" }, "provider\_guid": "{Removed}", "computer\_name": "Removed", "thread\_id": 5056, "host": "Removed", "source\_name": "Microsoft-Windows-Security-Auditing", "@version": "1", "task": "Anmelden", "event\_data": { "TransmittedServices": "-", "TargetUserSid": "Removed", "SubjectLogonId": "0x0", "KeyLength": "0", "SubjectUserSid": "S-1-0-0", "SubjectDomainName": "-", "ProcessName": "-", "IpPort": "49216", "AuthenticationPackageName": "Kerberos", "LogonType": "3", "LmPackageName": "-", "TargetDomainName": "Removed", "IpAddress": "Removed", "TargetLogonId": "0xa46c7b68", "ProcessId": "0x0", "TargetUserName": "Removed", "LogonGuid": "{2338B7DD-65B0-3DEC-5787-ABDCAAA843EC}", "LogonProcessName": "Kerberos", "SubjectUserName": "-" }, "record\_number": "744583588", "@timestamp": "2018-04-09T08:29:48.865Z", "log\_name": "Security" }, "fields": { "@timestamp": [ "2018-04-09T08:29:48.865Z" ] }, "sort": [ 1523262588865 ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 10, 2018, 2:29pm UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/2 "2018-04-10T14:29:47Z")
> but any filters I tried did not match, just two not working examples > if "berwachung erfolgreich" in [keywords] { > drop { } > } The `keywords` array doesn't contain the string "berwachung erfolgreich". Have you tried "Überwachung erfolgreich" instead? > if [keywords] =~ /^.berwachung erfolgreich./ { > drop { } > } You can't perform regexp matches against an array. > if "berwachung erfolgreich" in [\_source][keywords] { > drop { } > } Your event doesn't have a `_source` field. ---
**Author:** ![sru](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sru](https://discuss.elastic.co/u/sru)\ **Post date:** [April 11, 2018, 5:29am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/3 "2018-04-11T05:29:40Z")
Hi Magnus thank you for your answer, tried your suggestion but did not get success yet. above are two sample kibana json outputs, one of them seemed to me as it is in \_source but I might be wrong in that. I did retry this three filters for testing: if "Überwachung erfolgreich" in [keywords] { drop { } } if "\*berwachung erfolgreich" in [keywords] { drop { } } if "\xC3\x9Cberwachung erfolgreich" in [keywords] { drop { } } the last one is based on the output on the console, which I added here as well, none of them was successfull. Apr 11 06:51:54 host-replaced logstash: "beat" =\> { Apr 11 06:51:54 host-replaced logstash: "name" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "hostname" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "version" =\> "6.2.3" Apr 11 06:51:54 host-replaced logstash: }, Apr 11 06:51:54 host-replaced logstash: "host" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "record\_number" =\> "712930379", Apr 11 06:51:54 host-replaced logstash: "event\_data" =\> { Apr 11 06:51:54 host-replaced logstash: "SubjectUserSid" =\> "S-1-0-0", Apr 11 06:51:54 host-replaced logstash: "SubjectUserName" =\> "-", Apr 11 06:51:54 host-replaced logstash: "SubjectDomainName" =\> "-", Apr 11 06:51:54 host-replaced logstash: "IpAddress" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "TargetUserSid" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "TransmittedServices" =\> "-", Apr 11 06:51:54 host-replaced logstash: "AuthenticationPackageName" =\> "Kerberos", Apr 11 06:51:54 host-replaced logstash: "ProcessId" =\> "0x0", Apr 11 06:51:54 host-replaced logstash: "LogonGuid" =\> "{Removed}", Apr 11 06:51:54 host-replaced logstash: "LmPackageName" =\> "-", Apr 11 06:51:54 host-replaced logstash: "IpPort" =\> "59413", Apr 11 06:51:54 host-replaced logstash: "LogonType" =\> "3", Apr 11 06:51:54 host-replaced logstash: "ImpersonationLevel" =\> "%%1833", Apr 11 06:51:54 host-replaced logstash: "TargetDomainName" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "SubjectLogonId" =\> "0x0", Apr 11 06:51:54 host-replaced logstash: "TargetLogonId" =\> "0x72a51ab4", Apr 11 06:51:54 host-replaced logstash: "TargetUserName" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "KeyLength" =\> "0", Apr 11 06:51:54 host-replaced logstash: "LogonProcessName" =\> "Kerberos", Apr 11 06:51:54 host-replaced logstash: "ProcessName" =\> "-" Apr 11 06:51:54 host-replaced logstash: }, Apr 11 06:51:54 host-replaced logstash: "@version" =\> "1", Apr 11 06:51:54 host-replaced logstash: "computer\_name" =\> "Removed", Apr 11 06:51:54 host-replaced logstash: "thread\_id" =\> 1436, Apr 11 06:51:54 host-replaced logstash: "provider\_guid" =\> "{Removed}", Apr 11 06:51:54 host-replaced logstash: "event\_id" =\> 4624, Apr 11 06:51:54 host-replaced logstash: "log\_name" =\> "Security", Apr 11 06:51:54 host-replaced logstash: "opcode" =\> "Info", Apr 11 06:51:54 host-replaced logstash: "source\_name" =\> "Microsoft-Windows-Security-Auditing", Apr 11 06:51:54 host-replaced logstash: "level" =\> "Informationen" Apr 11 06:51:54 host-replaced logstash: } Apr 11 06:51:56 host-replaced logstash: { Apr 11 06:51:56 host-replaced logstash: "process\_id" =\> 584, Apr 11 06:51:56 host-replaced logstash: "computer\_name" =\> "Removed", Apr 11 06:51:56 host-replaced logstash: "thread\_id" =\> 5604, Apr 11 06:51:56 host-replaced logstash: "task" =\> "Anmelden", Apr 11 06:51:56 host-replaced logstash: "keywords" =\> [ Apr 11 06:51:56 host-replaced logstash: [0] "\xC3\x9Cberwachung erfolgreich" Apr 11 06:51:56 host-replaced logstash: ], Apr 11 06:51:56 host-replaced logstash: "provider\_guid" =\> "{Removed}", Apr 11 06:51:56 host-replaced logstash: "@timestamp" =\> 2018-04-11T04:51:52.664Z, Apr 11 06:51:56 host-replaced logstash: "type" =\> "wineventlog", Apr 11 06:51:56 host-replaced logstash: "event\_id" =\> 4624, ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 11, 2018, 6:04am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/4 "2018-04-11T06:04:17Z")
Works fine here. ```nohighlight $ cat test.config input { stdin { codec => json_lines } } output { stdout { codec => rubydebug } } filter { if "Überwachung erfolgreich" in [keywords] { drop { } } } $ cat data {"keywords": ["Überwachung erfolgreich"]} $ /opt/logstash/bin/logstash -f test.config < data Settings: Default pipeline workers: 8 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} ``` ---
**Author:** ![sru](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sru](https://discuss.elastic.co/u/sru)\ **Post date:** [April 11, 2018, 6:46am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/5 "2018-04-11T06:46:39Z")
using the same setup as you and get an output. /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ -f windows.conf \< data Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties { "host" =\> "Removed", "@timestamp" =\> 2018-04-11T06:44:40.956Z, "keywords" =\> [ [0] "\xC3\x9Cberwachung erfolgreich" ], "@version" =\> "1" } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 11, 2018, 7:27am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/6 "2018-04-11T07:27:29Z")
What's the output of `locale`? It looks like you're not running a UTF-8-based locale. ---
**Author:** ![sru](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sru](https://discuss.elastic.co/u/sru)\ **Post date:** [April 11, 2018, 7:27am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/7 "2018-04-11T07:27:56Z")
locale LANG=en\_US.UTF-8 LC\_CTYPE="en\_US.UTF-8" LC\_NUMERIC="en\_US.UTF-8" LC\_TIME="en\_US.UTF-8" LC\_COLLATE="en\_US.UTF-8" LC\_MONETARY="en\_US.UTF-8" LC\_MESSAGES="en\_US.UTF-8" LC\_PAPER="en\_US.UTF-8" LC\_NAME="en\_US.UTF-8" LC\_ADDRESS="en\_US.UTF-8" LC\_TELEPHONE="en\_US.UTF-8" LC\_MEASUREMENT="en\_US.UTF-8" LC\_IDENTIFICATION="en\_US.UTF-8" LC\_ALL= ---
**Author:** ![sru](https://avatars.discourse-cdn.com/v4/letter/s/b9e5f3/32.png) [@sru](https://discuss.elastic.co/u/sru)\ **Post date:** [April 11, 2018, 7:28am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/8 "2018-04-11T07:28:44Z")
java -version java version "1.8.0\_152" Java(TM) SE Runtime Environment (build 1.8.0\_152-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 11, 2018, 8:26am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/9 "2018-04-11T08:26:37Z")
Okay, weird. Then I don't know what the problem is. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 9, 2018, 8:26am UTC](https://discuss.elastic.co/t/winlogbeat-keywords-filtering-issue-in-logstash/127278/10 "2018-05-09T08:26:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.