Logstash filter not wroking

Elastic Stack Logstash
1

Hello

I'm new with logstash and i'm having problems in filter condition, i would like to ship onlt llogs/lines with ERROR, DEBUG, INFO and WARN message.

Here's the working config only one string on if condition.

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}
 if "DEBUG" not in [message] {

}
else {
drop { }
}
}

not working A:

 if "DEBUG" not in [message] {

}
else if "WARN" not in [message] {
}
else if "INFO" not in [message] {
}
else if "ERROR" not in [message] {
}
else {
drop { }
}
}

not working B:

if "DEBUG" not in [message] or "WARN" not in [message] or "INFO" not in [message] or "ERROR" not in [message] {

2

In example B you need to use the and operator, not or. Otherwise you'll drop all messages except those containing DEBUG, WARN, INFO and ERROR at the same time.

3

hello, thanks for the prompt response, my goal is to drop all the messages without DEBUG, WARN, INFO and ERROR.

4

example message withour DEBUG, INGO, ERROR, WARN

{
"_index": "maprouter-2017.09.14",
"_type": "log",
"_id": "AV5_TmTHj1PRrJFkBAHk",
"_version": 1,
"_score": 1,
"_source": {
"message": "00: 01 00 03 03 00 00 00 08 -- -- -- -- -- -- -- -- | ........ ",
"@version": "1",
"@timestamp": "2017-09-14T07:33:14.194Z",
"type": "log",
"input_type": "log",
"count": 1,
"beat": {
"hostname": "ip-10-3-101-15",
"name": "ip-10-3-101-15"
},
"source": "/opt/esc/logs/cmaprouter101.log",
"offset": 1592332,
"fields": null,
"host": "ip-10-3-101-15",
"tags": [
"beats_input_codec_plain_applied"
]
},
"fields": {
"@timestamp": [
1505374394194
]
}
}

5

my goal is to drop all the messages without DEBUG, WARN, INFO and ERROR.

Yes, I know.

example message withour DEBUG, INGO, ERROR, WARN

For that example message your option B works:

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  if "DEBUG" not in [message] or "WARN" not in [message] or "INFO" not in [message] or "ERROR" not in [message] {
    drop { }
  }
}
$ echo '00: 01 00 03 03 00 00 00 08 -- -- -- -- -- -- -- -- | ........' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

(Logstash isn't emitting any events, proving that the drop filter worked.)

However, for other kinds of messages you do need to follow the advice I gave earlier.

6

I've tried to use and operator, and it works on my testing. But on the actual setup i Dont see any data on my ES/Kibana :(.

$ echo '00: 01 00 03 03 00 00 00 08 -- -- -- -- -- -- -- -- | ........' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 1
Logstash startup completed
Logstash shutdown completed

$ echo ' DEBUG o.m.protocols.sctp.AssociationImpl - Rx : Ass=SAP_cmap101 PayloadData [dataLength=8, complete=true, unordered=true, payloadProtocolId=3, streamNumber=0, data=' | /opt/logstash/bin/logstash -f test.config

Settings: Default pipeline workers: 1
Logstash startup completed
{
"message" => " DEBUG o.m.protocols.sctp.AssociationImpl - Rx : Ass=SAP_cmap101 PayloadData [dataLength=8, complete=true, unordered=true, payloadProtocolId=3, streamNumber=0, data=",
"@version" => "1",
"@timestamp" => "2017-09-14T09:05:51.249Z",
"host" => "ip-10-3-2-105"
}
Logstash shutdown completed

7

it's now working!!!! thank you so much,

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.