Hot-Warm Architecture - Storage Handling

[jorj (George)]

Hello Christian

Thank you for your quick response,

@Christian_Dahlqvist

As explained in this blog post, each shard comes with some amount of overhead in terms of heap usage. Since heap is finite, there is a limit to how much data a node can hold, and this depends on the type of data as well as the mappings. How much data you can store on a node therefore often becomes and exercise in optimising heap usage. You will need to benchmark to see how much data each of your nodes can hold, but in my experience 60TB sounds far too much. I would expect you to require a significantly larger number of warm nodes to handle that data volume.

Regarding point 1, even if by doing a shrinking and 1 segment configuration (as per the Hot-Warm best practice), it will still be the same scenario?
Noting that the main requirement of this design is to be able to search old logs (and export the results) based on a Time-range and Source/Destination IP.

It's a CGNAT event log collection

Regards,
Jorj