we all run into the issue of running out of space on your initial ES instance. I launched a second and it spooled up but when i look at the summary for my instance it says:
1 primary and 1 replica.
If i recall, my first instance was a combined master/data node. My second one was just a data node.
I was hoping that by adding this second instance, which points to another 4 TB of space to use, it would fill that up slowly.
What I think i am observing is that the 2 instances arent acting as such. I think the second instance is a clone of the first. I noticed they had the same size.
What did I do wrong? I presume it would try to balance the data, maybe shift some from A to B while continuing to index, which frees up space on A. I havent noticed that happening though.
Is it because 1 is the Master/Data Combo and the other is Data?
I am in Kibana at the moment, and i was was looking at the management for the index, and im thinking of how I could edit it. I didnt see in the index management though that I had 2 ES instances up and running other than the Summary telling me 1 primary and 1 replica.
Also, I am not sure if this means anything, but it seems like for 1.4B records at the moment, i have only 1 shard, which i dont think is right. I have a 270G ( x 2 because of the above replica) shard. Seems that according to documentation, I should have shards that are under 40G or so. The issue I noticed though is that a shard is defined on init, so if i wanted to change this, I need to spin up a new index and migrate it all over. My issue is that I dont know how much my ingestion when complete will take up, so i cant necessarily backpedal to say: X/40g = Number of shards I need.
As a follow-up, should I do a single shard, full ingestion to find the space it takes, and then recreate the index with the right shards and then do a full dump from one index to another?