Drop filter not working

I have multiple log messages in a file which I am processing using logstash filter plugins. Then, the filtered logs are getting sent to elasticsearch.

There is one field called addID in a log message. I want to drop all the log messages which have a particular addID present. These particular addIDS are present in a ID.yml file.

Scenario: If the addID of a log message matches with any of the addIDs present in the ID.yml file, that log message should be dropped.

Could anyone help me in achieving this?

PLease help me Sir

Below is my config file.

input {

file {
   path => "/Users/jshaw/logs/access_logs.logs
   ignore_older => 0


filter {

grok {

    patterns_dir => ["/Users/jshaw/patterns"]
    match => ["message", "%{TIMESTAMP:Timestamp}+{IP:ClientIP}+{URI:Uri}"]


    field_split => "&?"
    include_keys => [ "addID" ]
    allow_duplicate_values => "false"


if [addID] in "/Users/jshaw/addID.yml" {


output {

     hosts => ["localhost:9200"]



You can't do a lookup like that unfortunately.

Thanks for replying @warkolm

Is there any workaround?

You might be able to use the translate filter, which you can then conditionally check if the translate was successful (value is equal to new replaced value, added a field, added a tag, etc). Doing a lookup on each incoming message against a large dictionary will be a performance bottleneck.


1 Like