Same Field name with different mappings

Elastic Stack Elasticsearch
1

Hi,

we have a system sending documents to ES with several fields having the same name, under different types, using different mappings. all in the same daily logstash index.
when we were running on 1.x all was working somehow...
now that we've upgraded to latest (2.2.0), ES rejects those fields from being indexed.

so I understand from documentation that we must have same mapping for fields with identical names.
assuming we cannot or dont want to do that, is there some way to work around it?

thanks,
Benjamin

2

Hi Benjamin. The work-around would be to use different indices for each type of data. While this kind of setup might have seemed to work in 1.x, it was a source of numerous bugs due to the fact thet when querying the whole index, elasticsearch would pick a mapping from a random type and assume it is fine to assume this mapping on all types. It could also cause elasticsearch to run into an unsupported Lucene configuration, like having different type of data (eg. strings and numerics) inder the same field name.

1 Like
3

thanks for the answer.
if indeed I use separate indices for each data type, should I expect any difficulties or anything special I need to do to make it work as usual with Kibana?
or as long as we keep the prefix same (logstash-*) it should be fine?

4

or as long as we keep the prefix same (logstash-*) it should be fine?

Yes, it should be fine. However, keep in mind that your indexes will be smaller so you may need to adjust the number of shards per index and/or start with weekly or monthly indexes instead of daily.

5

Hi,

thanks for the answer!

what's the rule of thumb for this kind of indices management?
I mean, currently I have the default shard configuration (so i guess its 5 shards.. with 1 replica).
what are the reasons to change it to less shards?
or what would be a good metric to consider for changing to weekly indices?

thanks,

6

I think the Designing for Scale chapter of The Definitive Guide will answer most of your questions.