Logstash Json Data processing issue

Elastic Stack Logstash
1

Hi ELK Community,

I need your help with an issue I'm encountering.

In my JSON data processing pipeline, some fields are coming in two formats: as an array and as a keyword.

When I set the mapping for "field_name" as an object in the index template, I get errors in the Logstash pipeline: "Unable to parse the keyword as object. Reason: Can't get text on a START_OBJECT at 1:6546."

When I set the mapping for "field_name" as a keyword in the index template, I receive the error: "Unable to parse the keyword as object, but found a concrete value."

How can I resolve this issue?

Thanks in advance!

2

Could you share the Elasticsearch mapping and 2 sample documents?

3

Mapping:
json.get.value ==> keyword
json.get.key ==> keyword

{
"@timestamp": "2024-10-06 11:09",
"json": {
"get": [
{
"value": "test-1",
"key": "check-1"
},
{
"value": "test-2",
"key": "check-2"
}
]
}
}
{
"@timestamp": "2024-10-06 11:09",
"json": {
"get": [
{
"value": "test-3",
"key": "check-3"
}
]
}
}

4

I don't see which one would fail.

5

As an array of concrete values or as an array of objects? It is no clear, the example you shared would not lead to conflict.

Can you share the entire log from Logstash when you get the error?

Also, you didn't share the mapping, you need to get the mapping using GET index-name/_mapping.

6

You can't have a field that is a keyword in some documents and an object in others. To resolve it you will need to either convert the field to an object when it is a keyword or convert the field to a keyword when it is an object. See this thread.

7

Thanks @badger,

I tried using the painless script but getting some errors.

Painless Script:

if(ctx.get.key != null)
{
ctx.get.key = ctx.get.key.toString();
}
if(ctx.get.value != null)
{
ctx.get.value = ctx.get.value.toString();
}

condition:
(ctx.get.key instanceof List) || (ctx.get.value instanceof List)

Error:"Illegal list shortcut value [key]."