Im currently working on pushing logs to elasticsearch using filebeat. In order to index these docs to the preferred location, i use an ingester node.
With this sample log, i use a series of processors to structure it. In one of the pipeline processors i use a script processor and do:
ctx._index = ctx.myIndex; ctx._type=ctx.myType;
This works but uses the internal version of elasticsearch. Since the doc contains a version of itself (myVersion field), i would like to use it so that i can make sure that an older version of what filebeat picks up does not override the doc if its an older one.
ive tried setting the ctx._version and addex ctx._version_type=external but i get an error when i do so (dynamic mapping exception).
How do i use my own versioning sequence with the pipelines?