Fighting with ingest pipeline create

[Russell_Fulton]

Version 7.14

I am trying to create a simple ingest pipeline in Kibana but can not figure out what syntax the condition will accept and there is no documentation that I can find.

I know how to specify what I want in the API but nothing I have tried in the condition box is accepted.

I get as far as "if" then it seems to want "()", and after that anything I type inside the () is flagged as an error.

what I want is if (field =='value')

I really wish there was a way of just entering the condition as it would appear in an API request!

[stephenb (Stephen Brown)]

Perhaps post your ingest pipeline and pseudo code of the condition where you want it?

Can't tell what you need

[Russell_Fulton]

I want

{
  "drop": {
    "if" : "<field_name> == '<value>'"
  }
}

which is the example in the manual. The problem is that I can't figure out what to put in the kibana "condition" box

[stephenb (Stephen Brown)]

Sorry still not clear...

What does your source document look like.

What is the value of you are trying to compare to?

I assume you looked like this and understand the ?

Example:

{
  "drop": {
     "if": "ctx.host?.name == 'Server1'"
  }
}

The conditions are pretty straightforward so I still must not be understanding something... == != equals not equals greater than etc.. or you can do contains etc.

[Russell_Fulton]

I am trying to do this from in Kibana, not the API. Kibana appears to use a different syntax which does not appear to be documented anywhere. The UI has a "smart" (dumb ? ; ) syntax checker that flags anything it does not like as soon as you type it. When you hover over the flag (~) it tells you what the possible things that can go there. Using this worked out it must start with if () but whatever I type next produces and error even though STRING is one of the options ???

As I said in my original post I can do this in the API by I want to use Kibana ( so that other, less technical people, can maintain it)

[stephenb (Stephen Brown)]

Apologies, I am confused which part of Kibana are you using...

Dev Tools? or Discover?

Not sure what "smart" syntax you are referring too, Kibana dev tools does do some but not much ... perhaps post a screenshot?

Here is a sample pipeline and test of it in Kibana -> Dev Tools

PUT /_ingest/pipeline/test-drop
{
  "processors" : [
    {
      "drop": {
        "if": "ctx?.object_type == 'disk'"
      }
    }
  ]
}


# This is how you simulated a pipeline
POST /_ingest/pipeline/test-drop/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "object_id": "1234",
        "object_type": "host"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "object_id": "2345",
        "object_type": "disk"
      }
    },
        {
      "_index": "index",
      "_id": "id",
      "_source": {
        "object_id": "3456",
        "object_type": "host"
      }
    }
  ]
}

and the result showing it is dropping the "disk" document

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "object_type" : "host",
          "object_id" : "1234"
        },
        "_ingest" : {
          "timestamp" : "2021-09-05T21:13:20.1752514Z"
        }
      }
    },
    null,
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "object_type" : "host",
          "object_id" : "3456"
        },
        "_ingest" : {
          "timestamp" : "2021-09-05T21:13:20.1752897Z"
        }
      }
    }
  ]
}

And as you can see I am doing this in Kibana 7.14

What am I missing? Maybe a screenshot of what you are seeing will help and the full actual code / pipeline.

EDIT Ahhh I guess you are using the Kibana - Stack Management - Ingest Node Pipelines GUI

(Sorry I rarely use that)

You can do full painless but if you just have simple conditions you just put in right hand side of the if block without the double quotes

    {
      "drop": {
        "if": "ctx?.object_type == 'disk'"
      }
    }

just put in

ctx?.object_type == 'disk'

[Russell_Fulton]

Ah! I am navigated this way:

home => management => ingest node pipeline => create

It had not occurred to me that there was more than one way of doing this in kibana.

Apologies for the disconnect!

[stephenb (Stephen Brown)]

No Worries... we were both assuming..

I did see the syntax "helper" yeah... a bit less than always helpful.

So what I usually do is create the pipeline in Dev Tools, then take a look in the GUI... see what it looks like... but mostly I like the Dev Tools

BUT The GUI does have a kinda cool way to test the pipeline sort of equivalent where you can put in your own test documents or actually
test a document from and existing index.