Max size of shard and index

Elastic Stack Elasticsearch
1

Hello,
I am new to Elastic search and started using elastic cloud services. I have the following questions, appreciate if some one can answer

  1. What is the max size of the shard? I have read in some place that its recommended to have this to 50GB. So, if I create an index which has default no of shards as 5, then max index size is 250GB. So, which means, if index size is approaching 250GB, should i create one more index?

  2. If I create multiple indexes, then can I query be fired on multiple indexes?

2

There is no theoretical limit to the GB size of a shard, you cannot go over 2 billion docs though.
But as you point out we do suggest keeping it under 50GB.

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/search-search.html#search-multi-index-type should cover the second question :slight_smile:

1 Like
3

Thanks a lot Mark Walkom.
If I design my solution like, would it be fine?

  1. Create a index with name "OurCompany-0-" with 5 shards. I watch its size over a period of time. If it crosses 250GB, then I can create one more index OurCompany-1-.

  2. Then I will issue the search on OurCompany-*

Is this is a recommended way of going about creating indexes.

1 Like
4

What sort of data is it?

5

Data would be mostly pdf document content. We would convert pdf to text and the text is the one we would add to the index.

A use case would be,
Search all the documents or records where the key word 'xyz' is there?
PDF file size can be around 50 MB on a average. If we convert all the PDF to text, associate it to a text field and store it.

Also, please guide me if I need to take any precautions where one field of the record would contain the text of the whole pdf (of say 50MB) ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.