Recommendation for multi index/node/cluster setup

[elitzur_e]

Hi. Recommendation for multi index/node/cluster setup

We are looking for the best way to approach this search scenario:

1 huge erp/crm with 100M~ documents.

Each document (word/pdf/img) belongs to a different customer

The request is to build an application that after authentication of a customer will allow that customer to search HIS documents.

Now, i believe that elasticsearch is the way to go here.

Though i am not sure what would be best strategy here.

i will replicate all the documents from the existing DB to be indexed to elastic.

Should i try to build 1 index per customer?(each customer has about 100k docs)

or have it separated in the node level? Would i need to have more than one cluster to have best results?

thanks for any direction or answer form your own experience.

Thanks

[DavidTurner]

@elitzur_e

Should i try to build 1 index per customer?(each customer has about 100k docs)

An index-per-customer design is normally a bad decision in the long run, because you will get more customers over time and eventually Elasticsearch will start to struggle with the number of indices that this entails. Also 100k documents is very small for an index. Here is an article with advice for shard sizing:

How many shards should I have in my Elasticsearch cluster?

@elitzur_e

or have it separated in the node level? Would i need to have more than one cluster to have best results?

I think you might be overthinking things. The simplest possible approach would be to have a single index for all the data, restricting each customer to their own documents using a bool query to add a term query to each search. Have you tried this? If yes, and it didn't work, where did it fall down?