Index.mapping.ignore_malformed for all new indices

Elastic Stack Elasticsearch
1

Hello all,

I have struggled with this for a few days now trying all the documentation and this forum without much luck...

I'm using Elastic Stack for a classic centralised logging use case. ES 5.2.2. Logstash is creating daily indices.

I have logs coming from many sources and I can't really control the format of each field so I'm running into field data type conflicts which by Elasticsearch defaults is handled by rejecting the whole document.

I have tried to use the the PUT mapping API to set "ignore_malformed": true for the fields that are causing issues but I keep getting unsupported parameters ignore_malformed errors.

What I can do is set index.mapping.ignore_malformed for new indices. That would be fine but I don't really want to do that manually for every index.

Question: Is there a way to set index.mapping.ignore_malformed as a default for all new indices?

Any answers and/or comments are appreciated :slight_smile:

Cheers,
AB

2

Seems like even if I create an index with "index.mapping.ignore_malformed": true I'm still seeing the same issue :confused:

GET logstash-2017.04.20.09/_settings
{
"logstash-2017.04.20.09": {
"settings": {
"index": {
"mapping": {
"ignore_malformed": "true"
},
"refresh_interval": "5s",
"number_of_shards": "5",
"provided_name": "logstash-2017.04.20.09",
"creation_date": "1492680822373",
"number_of_replicas": "1",
"uuid": "OjYYhYYUQbykVnwq7y5CUQ",
"version": {
"created": "5020299"
}
}
}
}
}

In my Elasticsearch logs I see this
[2017-04-20T09:54:52,605][DEBUG][o.e.a.b.TransportShardBulkAction] [es-03] [logstash-2017.04.20.09][4] failed to execute bulk item (index) index {log in JSON format}
java.lang.IllegalArgumentException: [severity] is defined as an object in mapping [my-logs] but this name is already used for a field in other types

Any ideas?

Cheers,
AB