Hello, So recently I started using es pivot transform. At every stage I am getting some incomplete data and using scripted metric I am completing it and sending it to the new index. Currently what this does is that every time a new document with same group by id comes in it updates the data in the new index. My question is that is there a way that instead of updating the document in the new index can I create a new document everytime?
"state" : "create",
"id" : "01",
"time" : "xyz" ,
"started" : True
It is by design that there is exactly one document corresponding to the given group-by id in the destination index. So if you use id field in group-by section on transform pivot config, the transform will do just this: take all the documents with the given id and summarize it in one document in the destination index.
If you'd like to have 1:1 relation between documents in source and destination, you should be using an id that is unique among documents, something like "event id" or maybe even "timestamp".
If you do not need continuous updates functionality, you could also try using reindex + scripted field. There is another ticket in which this is discussed:
I'm not sure about your particular use-case, but I could suggest one more option you might be interested in: ingest pipeline + script processor.
The script defined in the pipeline would be executed on each document without any grouping (so it won't override existing docs).
@przemekwitek Thank you for the reply. Let me elaborate a bit more on my use case. So, for my case a single event can have around 5 or more states(Unsure about the total number of events but it will be very high). Now, in the first state lets say it comes with 10 fields. After this state it will always come with fields which are either added or updated. So, the 2nd, 3rd and 4th states lets say comes with 4 fields and 5th with 6 field. My task is that every time a state comes in I have to create a docx with all the fields that have come in till that point and update the fields if any update has come in. So after every state there will be an extra docx in the new index.