# Elastic search Design - Room for improvement **URL:** \ **Category:** Elasticsearch\ **Created:** [May 22, 2020, 3:47am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842 "2020-05-22T03:47:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nages](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nages/32/46943_2.png) [@nages](https://discuss.elastic.co/u/nages)\ **Post date:** [May 22, 2020, 3:47am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/1 "2020-05-22T03:47:54Z")
We are starting designing a cluster and come up with the following optimal configuration . Please suggest if there is any scope for improvement or save some budget if it is over optimized. 1. 100 fields - 1 MB per field(including inverted index) - 125 MB( to be safer side) per document Total - 4M documents corresponds to 500 GB - 25GB per shard - 20 shards - total 40 shards including ( r = 1) - we had seen somewhere a shard size of 25GB works well in most of the scenarios. 2. Also it seems heap max 32 GB per each shard RAM) ( jvm uses compressed pointers ) works well - which translates to 64 GB (rest 50% for FS cache) .So , considering 256 GB RAM - this translates 2 shards per machine (128GB) - this translates 20 Data nodes per cluster(2 shards per each data node ) , 3 master nodes (HA ) , 1 coordinating node Please add your recommendations ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2020, 3:52am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/2 "2020-05-22T03:52:39Z")
Elasticsearch is generally not optimized for handling documents that large. Why are you documents that large? What do they contain? What is the use case? ---
**Author:** ![nages](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nages/32/46943_2.png) [@nages](https://discuss.elastic.co/u/nages)\ **Post date:** [May 22, 2020, 4:01am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/3 "2020-05-22T04:01:16Z")
do you see document of size 1MB is large considering in and around 100 fields ? and ES is not optimized for thse cases ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2020, 4:06am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/4 "2020-05-22T04:06:55Z")
I would consider documents over a few MB in size to be quite large and you mentioned them potentially being tens of MB in size, which I suspect potentially could be problematic and difficult to work with. This does sound like a quite unusual use case, so I would recommend you test and benchmark it to find out how well it works. ---
**Author:** ![nages](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nages/32/46943_2.png) [@nages](https://discuss.elastic.co/u/nages)\ **Post date:** [May 22, 2020, 5:44am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/5 "2020-05-22T05:44:22Z")
sorry i mean 100 MB per document ( not per field) - ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 22, 2020, 5:47am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/6 "2020-05-22T05:47:44Z")
I was talking about document size, not field size. What kind of data is this? What is the use case? How are you intending to search and use the results? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 19, 2020, 5:47am UTC](https://discuss.elastic.co/t/elastic-search-design-room-for-improvement/233842/7 "2020-06-19T05:47:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.