Ingest Pipeline, apply only if fields exist

[Tim_Rice]

Hello,

I have a ingest pipeline that does two things. It basically leverages kv and it applies geo info to a field.

PUT _ingest/pipeline/mitresplit
{
    "description": "splits technique_name and technique_id",
    "processors": [
      {
        "kv": {
          "field": "sfRuleName",
          "field_split": ",",
          "value_split": "="
        },
        "geoip": {
          "field": "sfDestinationIp",
          "target_field": "geo"
        }
      }
    ]
  }

And apply the pipeline

 PUT winlogbeat-*/_settings
 {
"index.default_pipeline":"_default"
 }

What I have figured out is if the document is sent in does NOT contain one of the field names, then it just dies.

How can I skin this cat? If either of the fields are missing, I want things to continue to work.

[Tim_Rice]

I think that I have figured it out by leveraging the below:

PUT _ingest/pipeline/mitresplit
{
"description": "splits technique_name and technique_id",
"processors": [
  {
    "kv": {
      "field": "event_data.RuleName",
      "field_split": ",",
      "value_split": "=",
      "ignore_failure" : true
    },
    "geoip": {
      "field": "event_data.DestinationIp",
      "ignore_failure" : true,
      "target_field": "geo"
    }
  }
]
}

PUT winlogbeat-*/_settings
{
"index.default_pipeline":"mitresplit"
}

[Christian_Dahlqvist]

Which version of Elasticsearch are you using?

[Tim_Rice]

6.6.0.

BTW, the

"ignore_failure" : true

Seems to have fixed the issue, but I don't like failures. It would be better (I think) to have a conditional statement that says, if field exists then do.

Thanks for the help.

[Christian_Dahlqvist]

Can you not use a conditional?