Elasticsearch Transforms

[przemekwitek (Przemysław Witek)]

Hi,
It is by design that there is exactly one document corresponding to the given group-by id in the destination index. So if you use id field in group-by section on transform pivot config, the transform will do just this: take all the documents with the given id and summarize it in one document in the destination index.
If you'd like to have 1:1 relation between documents in source and destination, you should be using an id that is unique among documents, something like "event id" or maybe even "timestamp".

If you do not need continuous updates functionality, you could also try using reindex + scripted field. There is another ticket in which this is discussed:

Reindex data with a new Field

There is an example of using reindex to modify the document in the docs here. It changes the name of a field but you can use the same script construct to do what you want to do. I suspect the script looks something like "script": "ctx._source['timestamp_hour'] = ctx._source['@timestamp'].getHourOfDay(). You are right that it will be much better at search time to use this new field. Watch the time zone, btw. I believe the hour that you get in this case is UTC. You could also do this with _update…