Ingest Pipelines - illegal_argument_exception reason field not present as part of path

Elastic Stack Elasticsearch
1

Hi
I trying apply a ingest pipeline into a datastream.
I'm using logstash to send to datastream, this datastream has a mapping:

 {
       "template": {
         "mappings": {
           "properties": {
             "@timestamp": {
               "type": "date",
               "format": "date_optional_time||epoch_millis"
             },
             "ipAddress": {
               "type": "ip"
             }
           }
         }
       }
     }

This is my Ingest Pipeline

name - logs-geoip

       {
         "geoip": {
           "field": "ipAddress"
         }
       }
     ]

When the log was arrived on elastic this pipeline not are invoking to start the fielld transformation ipAddress to put geoip information

I try run the command bellow to force run pipeline

POST logs-teste-geoip/_update_by_query?pipeline=logs-geoip

The response:

 {
   "took": 450,
   "timed_out": false,
   "total": 233549,
   "updated": 758,
   "deleted": 0,
   "batches": 1,
   "version_conflicts": 0,
   "noops": 0,
   "retries": {
     "bulk": 0,
     "search": 0
   },
   "throttled_millis": 0,
   "requests_per_second": -1,
   "throttled_until_millis": 0,
   "failures": [
     {
       "index": ".ds-logs-teste-2023.07.13-000001",
       "id": "vau7TIkBASEFwumKBSKz",
       "cause": {
         "type": "illegal_argument_exception",
         "reason": "field [ipAddress] not present as part of path [ipAddress]"
       },
       "status": 400
     }
     ...
     ...
     ...
   }

Some can help how to do this ingest pipeline works together with my datastream

I 'm using this doc to reference.

2

somebody can help me?

3

Hi @dmrlixos please try to be patient, there is a very limited number of folks answering questions right now...

You need to to provide a sample document.

And the result show 758 updated which leads me believe only a subset of your docs have the proper field.

Did you also try to run ingest _simulate to test your pipeline

4

Hi @stephenb i know man, sorry

About your question, make sense i'm looking into this issue with mo details and some cases, the ip address are IPV6 anothers case is a local ip like 192.168.x.x or 172.17.x.x etc.

I will still looking inside this error with more details to find the real way to do this.

Thanks for your support