Ingest pipeline : error if field does not exist - record missing

Elastic Stack Elasticsearch
1

Dear All,

my environment ELK 6.6.1
I have data with IP addresses which I want to load in a bulk. I wrote a ingest pipeline which converts IP to "geo_point". This works generally fine but if the record doesn't have this field I am missing this data completly in the elasticsearch database. So I add an "on_failure" but this doesn't help too. This is my filter:

PUT _ingest/pipeline/geoip
{
  "description": "Add geoip info",
  "processors": [
    {
      "geoip": {
        "field": "fortiip",
        "target_field": "geoip_tmp",
        "properties": [
          "location"
        ]
      }
    },
    {
      "set": {
        "field": "geoip",
        "value": "{{geoip_tmp.location.lat}}, {{geoip_tmp.location.lon}}"
      }
    },
    {
      "remove": {
        "field": "geoip_tmp"
      }
    }
  ],
  "on_failure": [
    {
      "set": {
        "field": "geoip",
        "value": "NULL"
      }
    }
  ]
}

The error message I see is:

{
      "index" : {
        "_index" : "spamcalcgrp",
        "_type" : "doc",
        "_id" : "100000154",
        "status" : 500,
        "error" : {
          "type" : "exception",
          "reason" : "java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [fortiip] not present as part of path [fortiip]",
          "caused_by" : {
            "type" : "illegal_argument_exception",
            "reason" : "java.lang.IllegalArgumentException: field [fortiip] not present as part of path [fortiip]",
            "caused_by" : {
              "type" : "illegal_argument_exception",
              "reason" : "field [fortiip] not present as part of path [fortiip]"
            }
          },
          "header" : {
            "processor_type" : "geoip"
          }
        }
      }
    },

Any ideas how I can avoid this issue.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.