Error converting field type in a reindexing

Elastic Stack Elasticsearch
,
1

Hello,

I'm having some visualization problems inside Elastic due to I'm sending all the data Beats collect to Logstash instead of directly to Elastisearch.
That made some fields change their type. For example, the field host.ip, when it's sent using Elasticsearch it's type IP, when in Logstash, it's type text, keyword. So these inconsistencies are creating some conflicts in my indexes.

I've been trying a lot of things I found in this forum and on the Internet and nothing is working at all. I'm trying to reindex my files, so I created a pipeline to do this while converting field types.

Here's the pipeline:

PUT _ingest/pipeline/convert_pipeline_auditbeat
{
  "processors": [
    {
      "convert": {
        "field": "client.ip",
        "type": "ip"
      }
    }
  ]
}

And here's the reindex I'm trying:

POST _reindex
{
  "source": {
    "index": "auditbeat-8.1.2-2022.04.18"
  },
  "dest": {
    "index": "auditbeat-8.1.2-2022.04.18_new",
    "pipeline": "convert_pipeline_auditbeat"
  }
}

But I'm getting the error "type" : "illegal_argument_exception", "reason" : "field [client] not present as part of path [client.ip]"

Any ideas to achieve this?

Thanks!

2

That is an Elasticsearch question, not about logstash, and you are more likely to get an answer if you move it to the Elasticsearch forum.

I would guess (I do not run Elasticsearch) that the fieldname has a . in it. logstash can distinguish between a field with a . in it ([client.ip]) and a field that contains another field ([client][ip]). I believe kibana and Elasticsearch use [client.ip] for both, leading to some ambiguity. I know that for a while Elasticsearch did not allow fieldnames to contain ., but that restriction was removed as the disambiguation code was improved.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.