Rename processor throwing error when trying to rename "level" to "log.level"

Elastic Stack Elasticsearch
1

I am trying to use the rename processor with field: "level" and target_field: "log.level." I am getting this error:

"error": {
          "type": "illegal_argument_exception",
          "reason": "cannot set [level] with parent object of type [java.lang.String] as part of path [log.level]"
        }

I found a very old issue that references the same problem: rename processor fails when trying to rename to nested field with same parent name · Issue #19892 · elastic/elasticsearch · GitHub

It appears this was fixed many years ago. Any ideas on what I'm doing wrong?

This is an example of what I'm trying to send:

{"@timestamp":"2024-08-09T03:24:13.709Z","container_name":"camera-service","source":"stdout","log":"{\"camera-service\":{\"service\":\"Repository\"},\"level\":\"debug\",\"message\":\"Finding by id 6681f65fbfbb2aa47f1be43e for collection gateways\",\"timestamp\":\"2024-08-09T03:24:13.709Z\"}"}

Here is my ingest pipeline:

[
  {
    "json": {
      "field": "log",
      "add_to_root": true
    }
  },
  {
    "rename": {
      "field": "level",
      "target_field": "log.level"
    }
  }
]

The json processor is working fine, and I see the parsed fields in the root of my document.

Thanks!

2

Can you remove the rename processor and share how your document looks like after the json processor?

Also, please specify which version you are running.

3

Looking at the linked issue and what you want to do, they are not the same problem.

The issue is about renaming a field into a nested field with the same parent name, as the example in issue shows, it is about being able to rename version into version.somethingElse.

What you want to do is to rename level into a nested field with a different parent name, so level into log.level.

If you change your rename processor to something like this, you will see that it works:

  {
    "rename": {
      "field": "level",
      "target_field": "level.type"
    }
  }

Your issue here is that you cannot rename level into log.level because the field log already exists in your document, and it is a string.

You have 2 options here:

  • Remove the log field before the rename processor.
  • Rename the log field to something else, also before the rename processor.

To remove the log field you can use:

      {
        "remove": {
          "field": "log"
        }
      }
4

I was looking at the output of just the json processor to post here and spotted the root issue that you brought up haha.

I tried to add the processors to either remove the log field or rename it before renaming the level field, but am getting this error for both:

{
  "error": {
    "type": "document_parsing_exception",
    "reason": "[1:229] failed to parse field [log] of type [text] in document with id 'y3KTNZEBT3Hv0nvc7Amv'. Preview of field's value: '{level=error}'",
    "caused_by": {
      "type": "illegal_state_exception",
      "reason": "Can't get text on a START_OBJECT at 1:213"
    }
  }
}

Here's are the two sets of processors I tried:

[
  {
    "json": {
      "field": "log",
      "add_to_root": true
    }
  },
  {
    "remove": {
      "field": [
        "log"
      ]
    }
  },
  {
    "rename": {
      "field": "level",
      "target_field": "log.level"
    }
  }
]

and

[
  {
    "json": {
      "field": "log",
      "add_to_root": true
    }
  },
  {
    "rename": {
      "field": "log",
      "target_field": "orig-log"
    }
  },
  {
    "rename": {
      "field": "level",
      "target_field": "log.level"
    }
  }
]

Thanks!

5

Ah yes, this would be expected because you already have documents in your index where the log field is a string, so you cannot have it as an object.

Did you create a template with a mapping for your index or are you using dynamic mapping and let Elasticsearch map the fields when it receives them?

To solve this you will need to create a new index, where the log field will be an object, or delete the current one.

6

Ah makes sense. Thanks so much for your help!