Drop document in ingest pipeline

[leochavez (Leo)]

Hi

I need to drop a document in an ingest pipeline , ie to not index it at all, I need to do it filtering the messages by the type of severity and discard them

I am testing it in the following way

PUT _ingest/pipeline/drop_model
{
  "processors": [
   {
      "drop": {
        "if": "ctx.severity == 'info'"
      }
    }
  ]
}

The logs in kibana that I need to discard are of the type

{
  "_index": "logstash",
  "_type": "events",
  "_id": "AWmh3Bauu--TaD0PBnfj",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-03-21T20:05:48.966480+00:00",
    "@version": "1",
    "host": "linux-009",
    "severity": "info",
    "facility": "user",
    "tag": "updatemgr",
    "pid": 0,
    "msg": "2019-03-21T20:05:48:946Z 'VcIntegrity' 139793736918784 INFO  [vcIntegrity, 1519] Getting IP Address from host name: linux-009"
  },
  "fields": {
    "@timestamp": [
      "2019-03-21T20:05:48.966Z"
    ]
  },
  "sort": [
    1553198748966
  ]
}

[Yogesh_Gaikwad]

Hi @leochavez,

Looks like you might not have drop processor for your ingest.
Could you please invoke following API and share the output?
GET /_nodes/ingest

The output should show you all the available processors in your setup.
Hope this helps.

Regards,
Yogesh Gaikwad

[leochavez (Leo)]

I can see the drop processor on my ingest, check:

 "ingest" : {
        "processors" : [
          {
            "type" : "append"
          },
          {
            "type" : "bytes"
          },
          {
            "type" : "convert"
          },
          {
            "type" : "date"
          },
          {
            "type" : "date_index_name"
          },
          {
            "type" : "dissect"
          },
          {
            "type" : "dot_expander"
          },
          {
            "type" : "drop"
          },
          {
            "type" : "fail"
          },
          {

[Yogesh_Gaikwad]

Hi @leochavez,

Do you see any error or it does the indexing? If any error could you please share.
Just a guess, Did you try with ctx.severity == \"info\"' instead of ctx.severity == 'info'?

Thank you.

I tried on my setup:-

PUT /_ingest/pipeline/my-pipe-1 '{ "processors" : [ { "drop" : { "if" : "ctx.sev_string == \"info\""  }  } ] }'

Try indexing doc with "info", verify response successful is 0, result noop

POST '/my-index/_doc/1?pipeline=my-pipe-1&pretty' '{ "sev_string" : "info", "severity": 5 }' -Q

{
  "_index" : "my-index",
  "_type" : "_doc",
  "_id" : "1",
  "_version" : -3,
  "result" : "noop",
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "failed" : 0
  }
}

Try indexing doc with "warn", verify response successful is 1, result created

POST '/my-index/_doc/2?pipeline=my-pipe-1&pretty' '{ "sev_string" : "warn", "severity": 5 }' -Q

{
  "_index" : "my-index",
  "_type" : "_doc",
  "_id" : "2",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Verify

GET /my-index/_search?pretty -Q

{
  "took" : 5,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-index",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "severity" : 5,
          "sev_string" : "warn"
        }
      }
    ]
  }
}