We have strict templates for our data but we are evolving and have some new data model ideas that we are working on experimenting with and need to have a way to perform an ETL type of operation from one cluster to another. I was wondering if something like this would be possible through a custom ingest plugin but I couldn't seem to easily figure out a way to change the shape of the ingest document. I was hoping I could do this on the fly as it would make moving data around and trial and error stuff easier now and in the future.
Is there a way to do this with a plugin or should I stick to a writing a separate app to do the ETL, or is there a better way that I'm not finding. I saw some documentation on the reindex API, but the model is rather large, and its going to be changing pretty drastically, so I wanted something a little easier to reuse going forward.