I have an index that receives a large amount of data. It is currently 250 GB in size (Elasticsearch 9.4.2), and data is continuously being ingested into it.
I want to migrate it to a data stream.
I also want to apply an ILM policy to ensure that the shard size does not exceed 50 GB.
What would happen to the existing large index if I managed to create a data stream? From that point on, we could rely on the ILM policy, which would create multiple backing indices with a single write index, ensuring that each backing index stays below 50 GB. However, the original 250 GB index would still remain, right?
This afternoon, I tried using an alias and this API:
what would it be the best strategy to complete this task?
Well, did it work? That looks like a pretty easy to convert if it works
The old indiex would remain, is it a single shard? Does it have managed retention? Is new data searched more than old data? If the old, large index becomes warm then cold, it being large shouldn't be that much of a problem.
yes, it worked in a separate env with smaller index(alias).
What we have implemented as a solution was to point Logstash to a new Data stream with a proper ILM policy.
We still have big index 250gb with only few shards (size beyond recommended up to 50gb per shard). We can delete it in few weeks.
Split API might be useful as well to migrate big index(with 1-2shards) to one with more shards.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
