Thanks for the response Igor_Motov. The problem isn't so much that the field could be mapped as float or text depending on which log message comes in first. That's what we're seeing in the system that says "Field Type Conflict" -- that is acceptable, and is not a problem. The problem is the other system where we actually get a "shards failed" error preventing us from querying any of the logs. Are you saying that "shards failed" is also normal/expected for elasticsearch when dynamic mapping is used and the data is not perfectly consistent?
Our scenario does call for using dynamic mapping. It might also be worth pointing out we are using the logstash json filter plugin to dynamically parse the fields.
I interpreted the above linked discussions (also this one) to mean that this inconsistent mapping, where one shard maps with one type and another shard maps with a different type, within the same index, should not happen since v2.x. E.g.:
On 2.0, this issue will be fixed as dynamic mappings will have to be validated on the master node first before being applied
and
Primary one thinks it's a String.
Primary two thinks it's a Date.
And you have kind of inconsistent mapping here.
That happened in the past. This is no longer true with recent versions.
It seems to be describing what we're seeing with v5.5.1 however, and it's not clear why (or how we can fix it so it consistently behaves like the 'good' system with the 'Field Type Conflict')