I need to figure out the best way of storing the information and I would appreciate your advice.
I am receiving 250,000,000 messages.
Each message contains:
- up to 7 traders
- up to 999 good items
- other data (but I simplify here)
Each trader has
- Identification number
Each good item has:
My users need to be able to look for traders, good items, ...
Basic example of query could be:
- list of traders involved in messages where good items type is ... and mass > ... and message issued by ... between date 1 and date 2 , and trader address "sound like" ..., group by trader type and location
In fact, users will be able to tackle the information from all directions (messages, traders, goods).
Therefore I thought storing the information the following way: => good items oriented
Per good item, I create a document that replicates all "searchable" information related to:
- the good item itself
- all the traders info
- the other searchable info from the message
The advantage of this solution would be to flattenize the data.
The drawback is the replication of the information.
That's why I would appreciate your expertise...
Many thanks in advance,