Im currently working on pushing logs to elasticsearch using filebeat. In order to index these docs to the preferred location, i use an ingester node.
{"name":"mydoc","myDocId":"myId1","myDocType":"someType","myIndex":"someIndex","myVersion":1}
With this sample log, i use a series of processors to structure it. In one of the pipeline processors i use a script processor and do:
...
ctx._index = ctx.myIndex; ctx._type=ctx.myType;
...
This works but uses the internal version of elasticsearch. Since the doc contains a version of itself (myVersion field), i would like to use it so that i can make sure that an older version of what filebeat picks up does not override the doc if its an older one.
ive tried setting the ctx._version and addex ctx._version_type=external but i get an error when i do so (dynamic mapping exception).
How do i use my own versioning sequence with the pipelines?