DROP processor does not appear to work in elastic search 6.5.0 with GROK

[ajazam1 (Abdul Jabbar Azam)]

Hello,

I have been trying to get the DROP processor to work in our Elastic Cloud. We are running v6.5.0.

I am using the following pipeline

 PUT _ingest/pipeline/pipeline1
  {
"description" : "logs",
"processors" : [
  {

    "set": {
      "field": "error",
      "value": ""
    },
    "grok" : {
      "field" : "message",
      "patterns" : [
        "%{GOT_ITEM_FROM:optype} %{WORD:tel}",
        "%{PROCESSED_ITEM_FROM:optype} %{WORD:tel}"
      ],
      "pattern_definitions" : {
        "GOT_ITEM_FROM" : "got item from",
        "PROCESSED_ITEM_FROM" : "processed item from"
      },
             "on_failure": [
                     {
                           "set" : {
                             "field": "error",
                             "value": "parseerror"
                           }
                   },
                   {
                    "drop" : {
                      "if": "true",
                      "tag": "inside drop processor"
                             }

                   }
             ]
    }
  }
  ]
  }

I am using the following to insert a document

POST filebeat/server/?pipeline=pipeline1
{

    "@timestamp": "2018-10-17T10:35:00.840Z",
    "host": {
      "name": "ISDN-SERVER"
    },
    "source": "C:\\Program Files (x86)\\2018_10_17.txt",
    "offset": 3172052,
    "message": "00:00:10.492(ID=0)(tid=10212)CelloProcessor_SMSProcessorQueueThread - (17) processed2 item from 1234",
    "input": {
      "type": "log"
    },
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "SERVER",
      "version": "6.4.2",
      "name": "SERVER"
    }

}

If I run

GET /filebeat/_search
{
"query": {
"match_all": {}
}
}

I get

{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "filebeat",
"_type" : "server",
"_id" : "k0vyZGcBpDcMS92QpBWK",
"_score" : 1.0,
"_source" : {
"offset" : 3172052,
"prospector" : {
"type" : "log"
},
"source" : """C:\Program Files (x86)\2018_10_17.txt""",
"message" : "00:00:10.492(ID=0)(tid=10212)CelloProcessor_SMSProcessorQueueThread - (17) processed2 item from 1234",
"error" : "parseerror",
"input" : {
"type" : "log"
},
"@timestamp" : "2018-10-17T10:35:00.840Z",
"host" : {
"name" : "SERVER"
},
"beat" : {
"hostname" : "SERVER",
"name" : "SERVER",
"version" : "6.4.2"
}
}
}
]
}
}

What I expected was for GROK to not do a pattern match and then for DROP to drop the document.

The DROP feature does work in logstash but we did not want to use logstash. It's another aws instance.

Has anybody got the DROP processor working in conjunction with GROK in Elastic Search without using logstash?

[spinscale (Alexander Reelsen)]

This looks like a bug to me. I opened an issue and you can follow it at https://github.com/elastic/elasticsearch/issues/36151