Filter don't filter and don't drop events in MongoDB log

Elastic Stack Logstash
1

Hi. I want to filter all events from MongoDB log except ACCESS component.
My config file is:

filter {
if [document_type] == "windows-mongodb" {
if [mongodb.log.component] != "ACCESS" {
drop{}
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\[%{WORD:mongodb.log.context}\] %{GREEDYDATA:mongodb.log.message}" }
}
}
}
I also tried:

if !( "ACCESS" in [message]) {
drop{}
}

And tried this:

if "ACCESS" not in [message]) {
drop{}
}

And in filebeat.yml I tried this:

exclude_lines: ['^!ACCESS']

And it don't work!! I get all components from the log.
Somebody can help me?

2

The test of mongodb.log.component comes before the grok that creates that field, so the test will never be true.

Also, field names containing . are not supported. They work most of the time, right up until they break things.

3

I put it after the grok too and it don't works. Also I changed it before to filed like "message" and it don't work

4

Show us an example message that Logstash didn't process correctly. Copy/paste from Kibana's JSON tab or use a stdout { codec => rubydebug } output.

5

The message is:

"message": "2018-09-02T08:28:03.144+0100 I NETWORK [conn3037863] end connection 212.18.253.12:61956 (53 connections now open)"

The conf file for logstash has:

if [document_type] == "windows-mongodb" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:mongodb.log.timestamp} %{WORD:mongodb.log.severity} %{WORD:mongodb.log.component} *\[%{WORD:mongodb.log.context}\] %{GREEDYDATA:mongodb.log.message}" }
}
if [mongodb.log.component] != "ACCESS" {
drop{}
}
}

In each filebeat.yml I added a new field document_type.

6

Please show he full message so we can see if document_type really contains "windows-mongodb" and whether there's a _grokparsefailure tag.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.