Max size of shard and index

Hello,
I am new to Elastic search and started using elastic cloud services. I have the following questions, appreciate if some one can answer

  1. What is the max size of the shard? I have read in some place that its recommended to have this to 50GB. So, if I create an index which has default no of shards as 5, then max index size is 250GB. So, which means, if index size is approaching 250GB, should i create one more index?

  2. If I create multiple indexes, then can I query be fired on multiple indexes?

There is no theoretical limit to the GB size of a shard, you cannot go over 2 billion docs though.
But as you point out we do suggest keeping it under 50GB.

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/search-search.html#search-multi-index-type should cover the second question :slight_smile:

1 Like

Thanks a lot Mark Walkom.
If I design my solution like, would it be fine?

  1. Create a index with name "OurCompany-0-" with 5 shards. I watch its size over a period of time. If it crosses 250GB, then I can create one more index OurCompany-1-.

  2. Then I will issue the search on OurCompany-*

Is this is a recommended way of going about creating indexes.

1 Like

What sort of data is it?

Data would be mostly pdf document content. We would convert pdf to text and the text is the one we would add to the index.

A use case would be,
Search all the documents or records where the key word 'xyz' is there?
PDF file size can be around 50 MB on a average. If we convert all the PDF to text, associate it to a text field and store it.

Also, please guide me if I need to take any precautions where one field of the record would contain the text of the whole pdf (of say 50MB) ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.