Hi
We are new to ES. We want to use it to power our hotel search feature. We have close to 80000 hotels/homes. After a lot of internal discussion, we decided to create date based indexes to properly handle our date availabilities for every property. This allows us to do a lot of other useful things while displaying search results to the customer.
So we have close to 500 indexes(for next 500 days) and 80000 documents per index.
We are still not sure to use Elastic Cloud or our own setup. For now, we have setup a 8 gb RAM instance on AWS.
To start, we decided to create 500 indexes with 1000 documents each. We tried both Bulk requests as well as individual requests. Update is really fast but after some time we are getting this error for some requests. Requests are not concurrent.
parent] Data too large, data for [<http_request>] would be [4122053212/3.8gb], which is larger than the limit of [4063657984/3.7gb], real usage: [4122039888/3.8gb], new bytes reserved: [13324/13kb]', u'type': u'circuit_breaking_exception', u'bytes_wanted': 4122053212, u'bytes_limit': 4063657984, u'durability': u'PERMANENT'}], u'reason': u'[parent] Data too large, data for [<http_request>] would be [4122053212/3.8gb], which is larger than the limit of [4063657984/3.7gb], real usage: [4122039888/3.8gb], new bytes reserved: [13324/13kb]
We tried upgrading the machine to 16GB, still the same issue.
Not sure which settings we should look at. Or is it that we are sending too many update requests which mess up our ES instance.
Help please. (We also tried Elastic Cloud trial which gives us 2 instances of 15GB, same issue there as well.)