If only the cost problem, I will address the matter, maybe I will query the data of the last 14 days only, but it is also considered big data, so it does not matter the cost if the search is fast, even if the RAM size reaches 1T
To answer the original question: we've demonstrated that a single node can query 1PiB of data. It wasn't fast, nor something that'd really be appropriate in a production environment, but nor was that really the limit for a single node in pure storage-capacity terms.
However, the storage costs alone for a dataset of this size will be over $1M per year, even before you start to think about RAM, CPU, network transit and miscellaneous other resource costs. At that kind of scale even tiny optimizations will have a massive impact on the total costs, and you will need to do some extremely careful analysis and design to choose the optimal path.
Yet, this is a free community forum, we can't reasonably do that level of analysis and design for you. If you're not comfortable with designing this kind of system yourself then I think you need to spend some of your $MM/yr budget on professional advice in this area rather than relying on the volunteers here. It'll save you money in the long run to do this properly.
2 Likes
I'll do that, for sure, but I'm in a comprehensive study phase now to assess what I need for that.
If the consultants you're planning to use need you to find out the answer to the question in this thread up front, I strongly recommend you find different consultants.
2 Likes