Similar problem to this discussion, but I was hoping for some input on what the best solution would be.
We're inputting json messages from multiple exchanges using the rabbitmq input plugin. Several of these types of messages use a "data" field, but for some "data" is a string and for others "data" is a dictionary of other objects (data.this:"hello", data.that:"goodbye").
This creates an error when elasticsearch can't index the data because it does not match the mapping created (depending on which type of message hit first after a new logstash index was created daily). The error is "org.elasticsearch.index.mapper.MapperParsingException: failed to parse [data]", for reference.
I know that we could define an index template and force "data" to be just a string, but I'm under the impression that would lock in the mappings - what happens if we start indexing a new message/log type with fields not included in that template? Updating the template every time is not a good long-term solution.
Can we just define the mapping for "data" and not include the other fields in the mapping and they will be created dynamically like normal? Does elasticsearch allow that? Barring that, the only solution I can think of is to filter the output so that the problematic message types go to their own separate indexes.
Thoughts or ideas? Am I missing something?