Container Resources : CPU = 2 Cores RAM= 16 GB
Prior we had 2 indexes (Events, Flows) each with 5 shard and 0 replica. We were writing 2000 (1000 Events + 1000 Flows) in Elasticsearch. CPU and RAM utilization kept at about 80-90 % all the time. Second, sometimes translog files increased to 100 GBs. We increased resources but situation remain same. Then I got a webinar of Bay Shanon, then I decided to use one shard in each index being on one node.
I just want to confirm that can elasticsearch handle the following load in one index with one shard on one node.
One Day Load (Approx.)
Events: 75 Million Documents = 75 GB FLows: 75 Million Documents = 75GB
One Month (Approx.)
Events: 1500 Million = 1500 GB FLows: 1500 Million = 1500 GB