Drop the complete message containing specific strings

Elastic Stack Logstash
1

Hello,

I have the following definition to drop messages from a log file containing strings and text:
input {
file {
path => "/opt/mapr/logs/cldb.log"
tags => "mapr_cldb"
codec => plain {charset => "ISO-8859-1"}
}
}

filter {
if "INFO" in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}
}

I see now, that all "WARNING" messages will also be dropped.
E.g:
2019-08-31 01:22:26,389 WARN Alarms [HB-2]: composeEmailMessage: Alarm raised:

What is the correct definition for this scenario?

2

Do you want to drop events that contain WARN? If not, why do have a line that does that?

3

Hello Badger,

I want to drop messages that contains the text:
"WARN log":
2019-07-05 07:09:49,102 WARN log [1813797132@qtp-2095303566-2]: EXCEPTION

"RpcProgram not found":
2018-10-18 05:20:13,7125 prog: 28, proc 180, RpcProgram not found

"The server has decided to close"
WARNING: The server has decided to close this client connection.

and so on. The WARNING above is not important while other WARNING messages are.

4

OK, so you need to make the pattern more specific. It might be as simple as

if " WARN log [" in [message] {

or you might have to use a regexp such as

if [message] =~ /^[-0-9]{10} [:,0-9]{12} WARN log \[/ {
5

The message
2019-08-19 01:22:26,389 WARN Alarms ......
is also dropped. And is not equal to "WARN log".

I reduced the amount of conf files to one and now the messages were forwarded as expected.
So I see an influence of filters defined in the conf files.
I tried it then with 2 conf files.
Each log/folder has its own conf file to filter the unwanted messages of this log:
File one:
[root@test logstash-2.2.2]# cat conf.d/mapr_apiserver.conf
input {
file {
path => "/opt/mapr/apiserver/logs/apiserver.log"
tags => "mapr_apiserver"
codec => plain {charset => "ISO-8859-1"}
}
}

filter {
if "INFO" in [message] { drop{ } }
if "WARN" in [message] { drop{ } }
if "Formatting information" in [message] { drop{ } }
if "Invalid parameters" in [message] { drop{ } }
if "fixation attacks" in [message] { drop{ } }
if "Continue searching" in [message] { drop{ } }
if "No data returned" in [message] { drop{ } }
if "No service found" in [message] { drop{ } }
if "ERROR AclCommands" in [message] { drop{ } }
if "sleep interrupted" in [message] { drop{ } }
}

File two:
[root@test logstash-2.2.2]# cat conf.d/mapr_cldb.conf
input {
file {
path => "/opt/mapr/logs/cldb.log"
tags => "mapr_cldb"
codec => plain {charset => "ISO-8859-1"}
}
}

filter {
if " INFO " in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}

As long as WARN messages will be dropped in mapr_apiserver, no WARN messages will be forwarded from cldb.log. If I delete the line and restart the agent, the WARN messages will be forwarded.
As I'm new with logstash, I may have it configured wrongly.

6

If path.config points to a directory then all the configuration files in that directory are combined into a single configuration. Events are read from all the inputs, sent through the filters defined in all of the files, and written to all of the defined outputs. So if one of your configuration files drops anything that contains "WARN" that will apply to all inputs. You may want to use multiple pipelines to apply input specific filters.

7

Yes you're right. I've tested it with:
input {
file {
path => "/opt/mapr/apiserver/logs/apiserver.log"
tags => "mapr_apiserver"
codec => plain {charset => "ISO-8859-1"}
type => "mapr_log"
}

file {
path => "/opt/mapr/logs/cldb.log"
tags => "mapr_cldb"
codec => plain {charset => "ISO-8859-1"}
type => "mapr_log"
}

}

filter {
if [tags] == "mapr_apiserver" {
if "INFO" in [message] { drop{ } }
if "WARN" in [message] { drop{ } }
if "Formatting information" in [message] { drop{ } }
if "Invalid parameters" in [message] { drop{ } }
if "fixation attacks" in [message] { drop{ } }
if "Continue searching" in [message] { drop{ } }
if "No data returned" in [message] { drop{ } }
if "No service found" in [message] { drop{ } }
if "ERROR AclCommands" in [message] { drop{ } }
if "sleep interrupted" in [message] { drop{ } }
}
else
if [tags] == "mapr_cldb" {
if " INFO " in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}
}

and now all WARN message in cldb.log were forwarded.
Is this coding suitable?

8

Yes, making the filters conditional based on the input is fine.

9

Thanks a lot

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.