Maximum Number of Indexes

Elastic Stack Elasticsearch
1

Hi,

I have just started exploring Elastic Search. I have a requirement similar to below thread

I want to store the data specific to brands for efficient search. I have around 4k brands and each brand has around 10k files. Initially I was thinking of creating one index per brand and store data. But looking at above link it is not efficient.

Second option was to create different index type for each brand. But I read somewhere that number of type should not be more than 5-6.

Can you tell me how to handle this use case.

Thanks in Advance
-Guru

2

Hello,

If you plan to store same kind of files for each brands, I suggest to create 1 index with 1 document type. Start with 5 shards and go ahead !

Big questions are:

  • what is for you "files" ?
  • What is the size of your files ?

bye,
Xavier

3

Thanks for the quick response.

Files would be pdf,msg,xls,eml etc. I am planning to use ingest attachment plugin to import these files to ES.
File size is 1-20 mb.

In my case the requirement is to search the data related to one brand at a time. Do you think creating only one index and storing all brand related data in the same index would decrease the performance?

Thank you,

-Guru

4

4 crore doc's is no a big deal for one index. But the issue here is file sizes (around 40 TB). I dont know how ES behaves with this big indexes.

5

Due to your file sizes, it'll create a very big indice if files are all at 20mb. In an other hand, having 4k indices (one per brand) will not be easy to administrate... Anyway, you'll have to store all this datas in one or more indices. Try to create one indice, add documents in, and monitore sizes. Add several nodes and monitore your platform :wink:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.