I've created a new elastic cloud deployment 7.14.x - Hot tier contains 1 node with 2 GB disc, Warm tier 1 node 320 GB disc. I wont do any document/index updating, a human will occasionally check logs via Kibana.
I've set up ILM so all indices are moved to warm tier immediately, then are deleted after 1 day. So move to warm tier after 0 days, then move to delete after 1 day, no snapshot. I confirmed all indices have the right ILM policy applied.
I'm sending logs at the rate about 60 MB per hour, about 20 thousand documents per hour spread over 3 indices (log-yyyy-MM, err-yyyy-MM, audit-yyyy-MM). About 24 hours have passed and hot tier disk is showing around 30% (~700 MB) disc being used and is still steadily rising, while the warm tier disk is 0% (~1.2 GB). Hot tier instance CPU is steady below 20%.
Hot tier disk usage is surprising me, especially the constant and steady increase. I expected it to be much lower since I've setup ILM to move the data immediately.
Am I doing something wrong? Am I wrong in expecting I can minimize Hot tier resource usage? How can I know when will hot tier disk usage stabilize? will it ever stabilize? What is the disk being spent on and why is it steadily increasing if data input amount is stable?
This is less than 5% of traffic I'll be sending to this cluster. I plan to have over 100 different indices coming in from around 50 beat applications and about 200 in-house developed log senders. And I'm trying to get an estimate for an appropriate node resource sizes.