Block certain strings from ingestion

[rahul_sirugudi]

Hi,

There are few back end applications which depend/listen to queues, if there are no messages in queue it will print as INFO level logs "NO MESSAGES AVAILABLE in queue". Is there a way where i can block this particular string to get ingested to Elasticsearch?

this is my flow looks like.
beats => logstash => es

[stephenb (Stephen Brown)]

Hi @rahul_sirugudi

Perhaps look at filebeat drop event processor

Or perhaps look at one of the many logstash topics on this...

Drop the complete message containing specific strings

Hello, I have the following definition to drop messages from a log file containing strings and text: input { file { path => "/opt/mapr/logs/cldb.log" tags => "mapr_cldb" codec => plain {charset => "ISO-8859-1"} } } filter { if "INFO" in [message] { drop{ } } if "[CLDB-1]:" in [message] { drop{ } } if "reqIncoming" in [message] { drop{ } } if "The server has decided to close" in [message] { drop{ } } if "WARN log" in [message] { drop{ } } if "RpcProgram not found" in [message] { dr…

[rahul_sirugudi]

Thanks, i tried from beats but now all the logs are blocked.

input {
  beats {
    port => 5044
    ssl  => false
  }
}
match => { "message" => "%{IPV4:ip} - \[%{TIMESTAMP_ISO8601:timestamp}\] - %{GREEDYDATA:message} - %{GREEDYDATA:pool} - %{LOGLEVEL:log-level} : %{GREEDYDATA:error-message}" }
output {
if [fields][type] == "test"
{
if "NO MESSAGES AVAILABLE in queue..." in [error-message] { drop { } }
{
stdout { codec => rubydebug }
elasticsearch {
hosts => ["0.0.0.0:9200"]
   user => "username"
   password => "password"
   index => "index"
}
}
}
}

[Ayush_Mathur]

@rahul_sirugudi Can you share what have you tried on beats to drop the events ?

[rahul_sirugudi]

oops my bad, i meant in the logstash beats config i made changes. { drop { } } . My guess is this will drop the mathced string.

[Ayush_Mathur]

@rahul_sirugudi I believe what you want to do can be done before even sending the event to logstash. As suggested by @stephenb please use the drop_event processor of filebeat putting in the required conditions - either equals or contains can be used in your case.

[rahul_sirugudi]

i am able to achieve this now thanks. My bad i should have kept my condition just after grok under filter section.

filter {
grok{
match => { "message" => "%{IPV4:ip} - \[%{TIMESTAMP_ISO8601:timestamp}\] - %{GREEDYDATA:message} - %{GREEDYDATA:pool} - %{LOGLEVEL:log-level} : %{GREEDYDATA:error-message}" }
}
if "NO MESSAGES AVAILABLE in queue..." in [error-message] { drop { } }
}

this helped me avoiding the ingesting messages.