I'm working on a data stream. I have created a test ILM policy. Have a couple of question regarding that.
Is there a huge difference time lag while retrieving data from hot, warm, cold phase indices. I tried inserting a few documents and used shrink API to reduce number of shards to 1 in the warm phase. But I'm not experiencing much time difference in retrieving the data from warm, hot and cold phases. Is that the case or will it differ when there is large amount of data. How much of a time difference can we expect for data retrieval between the phases.
I'm trying to see if there is a compression technology on data stream. ie, the data on the cold phase is not needed for searching anymore. Can we zip that data so that we can get more disk space on the cluster and can store store more data to the cold phase. Does the size reduces when we move index to cold phase itself? Or the shrink API is actually used for reducing the size of the index(I'm not sure if reducing the number of shards reduces the size of the index)