Hi, i need more deep explanation the difference between phase on ILM.
In documentation it is said:
hot : The index is actively being written to
warm: The index is generally not being written to, but is still queried
cold: The index is no longer being updated and is seldom queried. The information still needs to be searchable, but it’s okay if those queries are slower.
But there is no explanation what make it become slower or fast between phase. What happened in the background when indices move from one phase to another. And as far as I know, logs only written to the newest indices, right? Let say, I have 2 indices:
mylogs-2020-03-09 and mylogs-2020-03-08. It will write to indices mylogs-2020-03-09 even I didnt enable ILM?
Often you would setup dedicated parts of your cluster to handle each phase. This allows optimizing costs, for instance, you could have best IO on your hot tier and use a lower storage:memory ratio on hot than on warm.
btw, is it possible if we put hot,warm and cold index in 1 node. In my case, I only have 3 data node and I want to make hot,warm and cold index on each node.
if you only have 3 nodes and dedicate a node to each tier, you would be running without any replicas, meaning any kind of system issue, restart, upgrade etc., would mean some of the data are not available in the mean time. And if you should be unlucky to experience a storage issue, data will be gone. It is normally advisable to have two copies of all shards (i.e., one primary and one replica per shard).
You could have all 3 phases against the same set of nodes. In the warm phase, you could shrink and force merge the indices. The cold phase would further save memory by not eagerly loading the index into memory.
Ok, so i have been follow your advice. I create a policy, bind my index template to that. I create a policy where: when index size more than 2gb it will move to the next phase(to warm then cold). But i still confuses, how to check if my configuration is work?
if you index enough data into the associated alias, it should first trigger the rollover, then move to warm and then move to cold. You should be able to observe that indices are force merged to one shard in the warm phase and that they are frozen in the cold phase. You can see this in Kibana under "Index Management". Number of segments can be seen in the "stats" tab and the frozen status can be seen in the header (will say "Frozen" next to the name).
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.