Ingest Pipelines - illegal_argument_exception reason field not present as part of path

Hi
I trying apply a ingest pipeline into a datastream.
I'm using logstash to send to datastream, this datastream has a mapping:

 {
       "template": {
         "mappings": {
           "properties": {
             "@timestamp": {
               "type": "date",
               "format": "date_optional_time||epoch_millis"
             },
             "ipAddress": {
               "type": "ip"
             }
           }
         }
       }
     }

This is my Ingest Pipeline

name - logs-geoip

       {
         "geoip": {
           "field": "ipAddress"
         }
       }
     ]

When the log was arrived on elastic this pipeline not are invoking to start the fielld transformation ipAddress to put geoip information

I try run the command bellow to force run pipeline

POST logs-teste-geoip/_update_by_query?pipeline=logs-geoip

The response:

 {
   "took": 450,
   "timed_out": false,
   "total": 233549,
   "updated": 758,
   "deleted": 0,
   "batches": 1,
   "version_conflicts": 0,
   "noops": 0,
   "retries": {
     "bulk": 0,
     "search": 0
   },
   "throttled_millis": 0,
   "requests_per_second": -1,
   "throttled_until_millis": 0,
   "failures": [
     {
       "index": ".ds-logs-teste-2023.07.13-000001",
       "id": "vau7TIkBASEFwumKBSKz",
       "cause": {
         "type": "illegal_argument_exception",
         "reason": "field [ipAddress] not present as part of path [ipAddress]"
       },
       "status": 400
     }
     ...
     ...
     ...
   }

Some can help how to do this ingest pipeline works together with my datastream

I 'm using this doc to reference.

somebody can help me?

Hi @dmrlixos please try to be patient, there is a very limited number of folks answering questions right now...

You need to to provide a sample document.

And the result show 758 updated which leads me believe only a subset of your docs have the proper field.

Did you also try to run ingest _simulate to test your pipeline

Hi @stephenb i know man, sorry

About your question, make sense i'm looking into this issue with mo details and some cases, the ip address are IPV6 anothers case is a local ip like 192.168.x.x or 172.17.x.x etc.

I will still looking inside this error with more details to find the real way to do this.

Thanks for your support

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.