Hi, I'm pretty new to the ELK stack, but have spent some time in the docs, I'm truly amazed by all the features and existing (filter) plugins, but still somewhat unsure how to best put them to good use.
I'm looking for the best solution to fix attribute mapping: recently I've found that the 'dynamic mapping' caused some type mismatches:
- In some indeces the 'err' attribute was an object (e.g.
{id:"abc...", err: {code:"xyz..", message:"error message"} }) - In some indeces the 'err' attribute was just a string
Since this is a common problem, I've found that I need to need fix this by explict mapping and running an reindex, which i'm doing via POST _reindex.
To avoid the issue of 'inconsitent types' (resulting in IllegalArgumentExceptions or mapper_parsing_exceptions), I've also found that the script attribute can be passed to the _reindex method.
To avoid the issue for new entries (so not during an actual _reindex, but avoiding the issue beforehand), I've found I need to fix the attribute mapping, and this can be done in a couple of ways by using different filter plugins or even the native 'field references' [err]. Although these native reference doen't seem to support data type checking (string, object/hash, number, ...) as far as I can tell.
Now since these two solutions will have similar behavior (checking the type and casting it to an object/map); I would like to know if it's possible to solve them in the same language; I found that the reindex supports painless and groovy, for the filters there is a ruby plugin, but I would like to avoid writing this conversion in a lot of different languages, but rubyisn't supported by _reindex.
So I would like know:
- Am I missing an easier approach that doesn't require an extra language?
- Is there a single language that is supported for both the
filterplugins as well the_reindex?