Data too large circuit_breaking_exception

Elastic Stack Elasticsearch
1

Hi

We are new to ES. We want to use it to power our hotel search feature. We have close to 80000 hotels/homes. After a lot of internal discussion, we decided to create date based indexes to properly handle our date availabilities for every property. This allows us to do a lot of other useful things while displaying search results to the customer.

So we have close to 500 indexes(for next 500 days) and 80000 documents per index.

We are still not sure to use Elastic Cloud or our own setup. For now, we have setup a 8 gb RAM instance on AWS.

To start, we decided to create 500 indexes with 1000 documents each. We tried both Bulk requests as well as individual requests. Update is really fast but after some time we are getting this error for some requests. Requests are not concurrent.

parent] Data too large, data for [<http_request>] would be [4122053212/3.8gb], which is larger than the limit of [4063657984/3.7gb], real usage: [4122039888/3.8gb], new bytes reserved: [13324/13kb]', u'type': u'circuit_breaking_exception', u'bytes_wanted': 4122053212, u'bytes_limit': 4063657984, u'durability': u'PERMANENT'}], u'reason': u'[parent] Data too large, data for [<http_request>] would be [4122053212/3.8gb], which is larger than the limit of [4063657984/3.7gb], real usage: [4122039888/3.8gb], new bytes reserved: [13324/13kb]

We tried upgrading the machine to 16GB, still the same issue.

Not sure which settings we should look at. Or is it that we are sending too many update requests which mess up our ES instance.

Help please. (We also tried Elastic Cloud trial which gives us 2 instances of 15GB, same issue there as well.)

2

Unless you expect this to grow dramatically I do not see the point in creating so many tiny indices here. Each shard consumes a certain amount of overhead so this is quite inefficient. Try reindexing your data into a single index or use yearly indices if you feel you need time-based indices.

3

Thanks Christian.

We are using only one index now. And this issue disappeared. Indexing is fast as well.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.