What am i doing:
Parsing 50-100 GB of information per day from API (most part of information are duplicated, stored less than 1/1000 or even 1/10000 of information).
Is there any chance that ElasticSearch have a build-in alghoritm that can deliver information in real time to the clients with specific quary? If no, how would you do that?
PS Currently i'm in a planning stage of my development environment. Planning to use Node.js + MongoDB (for static) + ElasticSearch, but i might change it if there are better way to implement that feature.