Block certain strings from ingestion

Elastic Stack Elasticsearch
1

Hi,

There are few back end applications which depend/listen to queues, if there are no messages in queue it will print as INFO level logs "NO MESSAGES AVAILABLE in queue". Is there a way where i can block this particular string to get ingested to Elasticsearch?

this is my flow looks like.
beats => logstash => es

2

Hi @rahul_sirugudi

Perhaps look at filebeat drop event processor

Or perhaps look at one of the many logstash topics on this...

1 Like
3

Thanks, i tried from beats but now all the logs are blocked.

input {
  beats {
    port => 5044
    ssl  => false
  }
}
match => { "message" => "%{IPV4:ip} - \[%{TIMESTAMP_ISO8601:timestamp}\] - %{GREEDYDATA:message} - %{GREEDYDATA:pool} - %{LOGLEVEL:log-level} : %{GREEDYDATA:error-message}" }
output {
if [fields][type] == "test"
{
if "NO MESSAGES AVAILABLE in queue..." in [error-message] { drop { } }
{
stdout { codec => rubydebug }
elasticsearch {
hosts => ["0.0.0.0:9200"]
   user => "username"
   password => "password"
   index => "index"
}
}
}
}
4

@rahul_sirugudi Can you share what have you tried on beats to drop the events ?

5

oops my bad, i meant in the logstash beats config i made changes. { drop { } } . My guess is this will drop the mathced string.

6

@rahul_sirugudi I believe what you want to do can be done before even sending the event to logstash. As suggested by @stephenb please use the drop_event processor of filebeat putting in the required conditions - either equals or contains can be used in your case.

7

i am able to achieve this now thanks. My bad i should have kept my condition just after grok under filter section.

filter {
grok{
match => { "message" => "%{IPV4:ip} - \[%{TIMESTAMP_ISO8601:timestamp}\] - %{GREEDYDATA:message} - %{GREEDYDATA:pool} - %{LOGLEVEL:log-level} : %{GREEDYDATA:error-message}" }
}
if "NO MESSAGES AVAILABLE in queue..." in [error-message] { drop { } }
}

this helped me avoiding the ingesting messages.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.