I have the following definition to drop messages from a log file containing strings and text:
input {
file {
path => "/opt/mapr/logs/cldb.log"
tags => "mapr_cldb"
codec => plain {charset => "ISO-8859-1"}
}
}
filter {
if "INFO" in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}
}
I see now, that all "WARNING" messages will also be dropped.
E.g:
2019-08-31 01:22:26,389 WARN Alarms [HB-2]: composeEmailMessage: Alarm raised:
The message
2019-08-19 01:22:26,389 WARN Alarms ......
is also dropped. And is not equal to "WARN log".
I reduced the amount of conf files to one and now the messages were forwarded as expected.
So I see an influence of filters defined in the conf files.
I tried it then with 2 conf files.
Each log/folder has its own conf file to filter the unwanted messages of this log:
File one:
[root@test logstash-2.2.2]# cat conf.d/mapr_apiserver.conf
input {
file {
path => "/opt/mapr/apiserver/logs/apiserver.log"
tags => "mapr_apiserver"
codec => plain {charset => "ISO-8859-1"}
}
}
filter {
if "INFO" in [message] { drop{ } }
if "WARN" in [message] { drop{ } }
if "Formatting information" in [message] { drop{ } }
if "Invalid parameters" in [message] { drop{ } }
if "fixation attacks" in [message] { drop{ } }
if "Continue searching" in [message] { drop{ } }
if "No data returned" in [message] { drop{ } }
if "No service found" in [message] { drop{ } }
if "ERROR AclCommands" in [message] { drop{ } }
if "sleep interrupted" in [message] { drop{ } }
}
filter {
if " INFO " in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}
As long as WARN messages will be dropped in mapr_apiserver, no WARN messages will be forwarded from cldb.log. If I delete the line and restart the agent, the WARN messages will be forwarded.
As I'm new with logstash, I may have it configured wrongly.
If path.config points to a directory then all the configuration files in that directory are combined into a single configuration. Events are read from all the inputs, sent through the filters defined in all of the files, and written to all of the defined outputs. So if one of your configuration files drops anything that contains "WARN" that will apply to all inputs. You may want to use multiple pipelines to apply input specific filters.
filter {
if [tags] == "mapr_apiserver" {
if "INFO" in [message] { drop{ } }
if "WARN" in [message] { drop{ } }
if "Formatting information" in [message] { drop{ } }
if "Invalid parameters" in [message] { drop{ } }
if "fixation attacks" in [message] { drop{ } }
if "Continue searching" in [message] { drop{ } }
if "No data returned" in [message] { drop{ } }
if "No service found" in [message] { drop{ } }
if "ERROR AclCommands" in [message] { drop{ } }
if "sleep interrupted" in [message] { drop{ } }
}
else
if [tags] == "mapr_cldb" {
if " INFO " in [message] { drop{ } }
if "[CLDB-1]:" in [message] { drop{ } }
if "reqIncoming" in [message] { drop{ } }
if "The server has decided to close" in [message] { drop{ } }
if "WARN log" in [message] { drop{ } }
if "RpcProgram not found" in [message] { drop{ } }
if "No such file or directory(2)" in [message] { drop{ } }
if "attempting to become a master" in [message] { drop{ } }
}
}
and now all WARN message in cldb.log were forwarded.
Is this coding suitable?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.