Processor foreach field error hash map

Elastic Stack Kibana
1

Hi,
I have this processor:

[
  {
    "foreach": {
      "field": "location",
      "processor": {
        "set": {
          "field": "geo.lat",
          "value": "{{_ingest._value.lat}}"
        }
      }
    }
  },
  {
    "foreach": {
      "field": "location",
      "processor": {
        "set": {
          "field": "geo.lon",
          "value": "{{_ingest._value.lon}}"
        }
      }
    }
  }
]

that should select a location field and split in two geo fields.

When I create the pipeline and test it I have this error:

[illegal_argument_exception] field [location] of type [java.util.HashMap] cannot be cast to [java.util.List]

The original document is:

{
  "_index": "device_data",
  "_type": "_doc",
  "_id": "3VX4onkBuzazNoZiQQmg",
  "_version": 1,
  "_score": 0,
  "fields": {
    "date": [
      "2021-05-25T12:00:49.000Z"
    ],
    "temp": [
      27.8
    ],
    "ver": [
      "0.15"
    ],
    "memFree": [
      49
    ],
    "busErr": [
      0
    ],
    "diskFree": [
      88
    ],
    "uptime": [
      1890480
    ],
    "location": [
      {
        "accuracy": [
          20
        ],
        "lon": [
          9.299167
        ],
        "lat": [
          45.516632
        ]
      }
    ],
    "id": [
      "NZOO-SNI-1-1"
    ],
    "ittver": [
      ""
    ],
    "avgLoad": [
      2.18
    ],
    "customer": [
      ""
    ]
  }
}
2

If there is only 1 location field per doc don't use foreach it is expecting an array of locations. The new JSON display in discover is a bit confusing at first glance but that looks like a single location field not an array of location fields.

I think you want something like this especially if you want geo to be a mappable geo_point

In this case convert functions as a set and sets the type as well

DELETE my-discuss-index

PUT my-discuss-index
{
  "mappings": {
    "properties": {
      "geo" : {"type": "geo_point"}
    }
  }
}

PUT _ingest/pipeline/my-discuss-pipeline
{
  "processors": [
    {
      "convert": {
        "field" : "location.lat",
        "target_field": "geo.lat",
        "type": "float"
      }
    },
        {
      "convert": {
        "field": "location.lon",
        "target_field": "geo.lon",
        "type": "float"
      }
    }
  ]
}

POST my-discuss-index/_doc?pipeline=my-discuss-pipeline
{
  "location": {
    "accuracy": 20,
    "lon": 9.299167,
    "lat": 45.516632
  }
}


GET my-discuss-index/_search
{
  "fields": [
    "*"
  ]
}

results

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my-discuss-index",
        "_type" : "_doc",
        "_id" : "5bJf0nkBX-iqkb9ijrVm",
        "_score" : 1.0,
        "_source" : {
          "geo" : {
            "lon" : 9.299167,
            "lat" : 45.516632
          },
          "location" : {
            "accuracy" : 20,
            "lon" : 9.299167,
            "lat" : 45.516632
          }
        },
        "fields" : {
          "geo" : [
            {
              "coordinates" : [
                9.299167,
                45.516632
              ],
              "type" : "Point"
            }
          ],
          "location.lat" : [
            45.516632
          ],
          "location.lon" : [
            9.299167
          ],
          "location.accuracy" : [
            20
          ]
        }
      }
    ]
  }
}
1 Like
3

The processor is perfect and it works.

But, I edited the settings "index.final_pipeline" of my index...and the new "geo" field is not added to the document...why?

Thanks

4

@stephenb any help?

5

How did you set final pipeline?

Do you see the pipeline settings in the actual index?

GET my-index-00001/_settings

{
  "my-index-00001" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "my-index-00001",
        "final_pipeline" : "geoip-info",  <!------ Here
        "creation_date" : "1623071886576",
        "number_of_replicas" : "1",
        "uuid" : "JprLcstMS-KQFTpYbz2MxA",
        "version" : {
          "created" : "7130099"
        }
      }
    }
  }
}
6

Now it works.

I was wrong not to convert lat and lon in geopoint.
Now he puts the markers in Maps but, I have a problem.
Any data containing "lat" and "lon" also has a "customer" field.
I would like, for each customer, to show me the last document (therefore the last location). While, if I click on the marker, it shows me the list of all documents with that customer ...

How can I do?

7

Please open a new thread with a good/ specific title... perhaps someone else can help.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.