Logstash trying to map object on empty text "Illegal argument exception STAT_OBJECT"

Elastic Stack Logstash
1

Hello,

I'm facing an ever lasting issue on some dataset where some "dynamic" plugins like kv or json can map certain field values as object.

For example service => "{}"

I'm not a big fan of per dataset fixes where i try to mutate the fields, i'm looking for a silver bullet if there is for this case !

I was thinking of a way to make logstash forgot about illegal fields so that it can index the rest of the document. I remember a mapping option like that.

2

This issue is not really a Logstash issue, logstash just logs the respons from Elasticsearch.

The problem here is that you are having mapping conflicts and when you try to index a document where one of the fields has a different mapping from the one in the index, elasticsearch will reject the document and logstash will drop it and send to dlq if enabled.

The mapping option you mentioned is probably the ignore_malformed setting, but this does not work for all mapping conflicts, in your specific case it will not work because your conflict seems to be related to fields that can be a object but are not mapped as an object.

In this case the document will always be dropped.

There is no silver bullet for this, you will need to fix it for every field in every dataset you have that is having mapping conflicts.

If the field is an object you need to create the correct mapping and if a document arrives where it is not an object, you need to remove the field or rename it.

1 Like
3

Yes indeed i was looking for a similar permissive option i'm loosing quite alot of data this way and i might need to rethink my indexing strategy with this issue because i will not be able to predict all the fields and all the possible evolution from different vendors...

For now remove_char_value and some gsub after json will do.

Thanks for your kind help.