Maximum Number of Indexes

[Guruprasad_Prabhu]

Hi,

I have just started exploring Elastic Search. I have a requirement similar to below thread

Maximum limit for indexes

Hi Jim Could you elaborate a bit on the overhead of indices? A single index-per-object is gives us a perfect setup for our dataflow, but how far can this scale reasonably? Though we're starting out at ~20,000 indices, I'd like to be able to plan for 1 million if possible. If this isn't too far fetched, which is the more important scaling factor, node size or node count? That is indeed far fetched Unless you plan on having hundreds of thousands of servers... However, fil…

I want to store the data specific to brands for efficient search. I have around 4k brands and each brand has around 10k files. Initially I was thinking of creating one index per brand and store data. But looking at above link it is not efficient.

Second option was to create different index type for each brand. But I read somewhere that number of type should not be more than 5-6.

Can you tell me how to handle this use case.

Thanks in Advance
-Guru

[xavierfacq]

Hello,

If you plan to store same kind of files for each brands, I suggest to create 1 index with 1 document type. Start with 5 shards and go ahead !

Big questions are:

• what is for you "files" ?
• What is the size of your files ?

bye,
Xavier

[Guruprasad_Prabhu]

Thanks for the quick response.

Files would be pdf,msg,xls,eml etc. I am planning to use ingest attachment plugin to import these files to ES.
File size is 1-20 mb.

In my case the requirement is to search the data related to one brand at a time. Do you think creating only one index and storing all brand related data in the same index would decrease the performance?

Thank you,

-Guru

[Ravi_Shanker_Reddy]

4 crore doc's is no a big deal for one index. But the issue here is file sizes (around 40 TB). I dont know how ES behaves with this big indexes.

[xavierfacq]

Due to your file sizes, it'll create a very big indice if files are all at 20mb. In an other hand, having 4k indices (one per brand) will not be easy to administrate... Anyway, you'll have to store all this datas in one or more indices. Try to create one indice, add documents in, and monitore sizes. Add several nodes and monitore your platform