# Facing error when trying to reindex from remote

**URL:** <https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116>\
**Category:** Elasticsearch\
**Created:** [November 26, 2018, 5:41am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116 "2018-11-26T05:41:06Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![pradla](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@pradla](https://discuss.elastic.co/u/pradla)\
**Post date:** [November 26, 2018, 5:41am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/1 "2018-11-26T05:41:06Z")

</div>

I am trying to reindex from remote and I am facing an error. This was running fine for a few days now but suddenly I am facing this error

my reindex code:

> curl -X POST "127.0.0.1:9201/\_reindex?pretty=true" -H 'Content-Type: application/json' -d'  
> {  
> "conflicts": "proceed",  
> "source": {  
> "remote": {  
> "host": "[http://localhost:9200](http://localhost:9200)"  
> },  
> "index": "my\_new\_twitter\_river"  
> },  
> "dest": {  
> "index": "twittertool2"  
> }  
> }'

Error I am getting:

> {  
> "error" : {  
> "root\_cause" : [  
> {  
> "type" : "circuit\_breaking\_exception",  
> "reason" : "[parent] Data too large, data for [\<http\_request\>] would be [7430139381/6.9gb], which is larger than the limit of [7345215897/6.8gb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=190/190b, accounting=7430139191/6.9gb]",  
> "bytes\_wanted" : 7430139381,  
> "bytes\_limit" : 7345215897  
> }  
> ],  
> "type" : "circuit\_breaking\_exception",  
> "reason" : "[parent] Data too large, data for [\<http\_request\>] would be [7430139381/6.9gb], which is larger than the limit of [7345215897/6.8gb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=190/190b, accounting=7430139191/6.9gb]",  
> "bytes\_wanted" : 7430139381,  
> "bytes\_limit" : 7345215897  
> },  
> "status" : 503  
> }

after some search I tried to increase the limit with this 🙂

> curl -XPUT localhost:9201/\_cluster/settings -H 'Content-Type: application/json' -d '{  
> "persistent" : {  
> "indices.breaker.fielddata.limit" : "75%"  
> }  
> }'

I got this acknowledgement:

> {"acknowledged":true,"persistent":{"indices":{"breaker":{"fielddata":{"limit":"75%"}}}},"transient":{}}

but the error is not resolved.

Does anyone has any insights on how to solve this?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 26, 2018, 6:23am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/2 "2018-11-26T06:23:44Z")

</div>

It's probably referring to `http.max_content_length`, [https://www.elastic.co/guide/en/elasticsearch/reference/6.5/modules-http.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/modules-http.html)

But be careful increasing it too large as it may cause an OOM.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 26, 2018, 6:33am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/3 "2018-11-26T06:33:14Z")

</div>

It sounds like you are suffering from heap pressure. You may need to scale up of out in order to add resources to the cluster or try to optimise your heap usage. What is the full output from the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)?

You should also be careful changing circuit breaker settings as pushing this too far can cause serious problems.

---

<div class="post-metadata">

**Author:** ![pradla](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@pradla](https://discuss.elastic.co/u/pradla)\
**Post date:** [November 26, 2018, 7:24am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/4 "2018-11-26T07:24:00Z")

</div>

Output from cluster stats api

> {  
> "\_nodes" : {  
> "total" : 1,  
> "successful" : 1,  
> "failed" : 0  
> },  
> "cluster\_name" : "twittertool-2.0",  
> "cluster\_uuid" : "KUcPiIQqT\_2OtgOfvxcCsw",  
> "timestamp" : 1543216908848,  
> "status" : "yellow",  
> "indices" : {  
> "count" : 2,  
> "shards" : {  
> "total" : 10,  
> "primaries" : 10,  
> "replication" : 0.0,  
> "index" : {  
> "shards" : {  
> "min" : 5,  
> "max" : 5,  
> "avg" : 5.0  
> },  
> "primaries" : {  
> "min" : 5,  
> "max" : 5,  
> "avg" : 5.0  
> },  
> "replication" : {  
> "min" : 0.0,  
> "max" : 0.0,  
> "avg" : 0.0  
> }  
> }  
> },  
> "docs" : {  
> "count" : 1212735146,  
> "deleted" : 260172548  
> },  
> "store" : {  
> "size" : "1.9tb",  
> "size\_in\_bytes" : 2150310164567  
> },  
> "fielddata" : {  
> "memory\_size" : "0b",  
> "memory\_size\_in\_bytes" : 0,  
> "evictions" : 0  
> },  
> "query\_cache" : {  
> "memory\_size" : "0b",  
> "memory\_size\_in\_bytes" : 0,  
> "total\_count" : 0,  
> "hit\_count" : 0,  
> "miss\_count" : 0,  
> "cache\_size" : 0,  
> "cache\_count" : 0,  
> "evictions" : 0  
> },  
> "completion" : {  
> "size" : "0b",  
> "size\_in\_bytes" : 0  
> },  
> "segments" : {  
> "count" : 762,  
> "memory" : "6.9gb",  
> "memory\_in\_bytes" : 7430139191,  
> "terms\_memory" : "6.5gb",  
> "terms\_memory\_in\_bytes" : 7045584686,  
> "stored\_fields\_memory" : "276.2mb",  
> "stored\_fields\_memory\_in\_bytes" : 289619488,  
> "term\_vectors\_memory" : "0b",  
> "term\_vectors\_memory\_in\_bytes" : 0,  
> "norms\_memory" : "3.6mb",  
> "norms\_memory\_in\_bytes" : 3847680,  
> "points\_memory" : "73.6mb",  
> "points\_memory\_in\_bytes" : 77275865,  
> "doc\_values\_memory" : "13.1mb",  
> "doc\_values\_memory\_in\_bytes" : 13811472,  
> "index\_writer\_memory" : "0b",  
> "index\_writer\_memory\_in\_bytes" : 0,  
> "version\_map\_memory" : "0b",  
> "version\_map\_memory\_in\_bytes" : 0,  
> "fixed\_bit\_set" : "0b",  
> "fixed\_bit\_set\_memory\_in\_bytes" : 0,  
> "max\_unsafe\_auto\_id\_timestamp" : 1542395973142,  
> "file\_sizes" : { }  
> }  
> },  
> "nodes" : {  
> "count" : {  
> "total" : 1,  
> "data" : 1,  
> "coordinating\_only" : 0,  
> "master" : 1,  
> "ingest" : 1  
> },  
> "versions" : [  
> "6.5.0"  
> ],  
> "os" : {  
> "available\_processors" : 40,  
> "allocated\_processors" : 40,  
> "names" : [  
> {  
> "name" : "Linux",  
> "count" : 1  
> }  
> ],  
> "mem" : {  
> "total" : "125gb",  
> "total\_in\_bytes" : 134291501056,  
> "free" : "21gb",  
> "free\_in\_bytes" : 22551023616,  
> "used" : "104gb",  
> "used\_in\_bytes" : 111740477440,  
> "free\_percent" : 17,  
> "used\_percent" : 83  
> }  
> },  
> "process" : {  
> "cpu" : {  
> "percent" : 1  
> },  
> "open\_file\_descriptors" : {  
> "min" : 1032,  
> "max" : 1032,  
> "avg" : 1032  
> }  
> },  
> "jvm" : {  
> "max\_uptime" : "1.3h",  
> "max\_uptime\_in\_millis" : 5038641,  
> "versions" : [  
> {  
> "version" : "1.8.0\_111",  
> "vm\_name" : "OpenJDK 64-Bit Server VM",  
> "vm\_version" : "25.111-b15",  
> "vm\_vendor" : "Oracle Corporation",  
> "count" : 1  
> }  
> ],  
> "mem" : {  
> "heap\_used" : "9.4gb",  
> "heap\_used\_in\_bytes" : 10097095392,  
> "heap\_max" : "9.7gb",  
> "heap\_max\_in\_bytes" : 10493165568  
> },  
> "threads" : 79  
> },  
> "fs" : {  
> "total" : "43.5tb",  
> "total\_in\_bytes" : 47897179586560,  
> "free" : "39.6tb",  
> "free\_in\_bytes" : 43624962523136,  
> "available" : "39.6tb",  
> "available\_in\_bytes" : 43624962523136  
> },  
> "plugins" : ,  
> "network\_types" : {  
> "transport\_types" : {  
> "security4" : 1  
> },  
> "http\_types" : {  
> "security4" : 1  
> }  
> }  
> }  
> }

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 26, 2018, 7:38am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/5 "2018-11-26T07:38:36Z")

</div>

It looks like you have very large shards (190GB) and that terms memory is taking up 6.5GB of heap space. I would recommend that you increase the heap size or scale out the cluster.

---

<div class="post-metadata">

**Author:** ![pradla](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@pradla](https://discuss.elastic.co/u/pradla)\
**Post date:** [November 26, 2018, 7:48am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/6 "2018-11-26T07:48:53Z")

</div>

Thank you!

I was kind of worried that I might end up with very huge shards but realized I have to fix them at the time of starting only! Is there a way to increase the number of shards to scale out given that now I have 2 TB of data?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 26, 2018, 7:57am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/7 "2018-11-26T07:57:01Z")

</div>

You do have the [split index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-split-index.html), but this requires your index to have been created with the `number_of_routing_shards` parameter set, which is currently not enabled by default. It may therefore require reindexing.

---

<div class="post-metadata">

**Author:** ![pradla](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@pradla](https://discuss.elastic.co/u/pradla)\
**Post date:** [November 26, 2018, 8:03am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/8 "2018-11-26T08:03:02Z")

</div>

Thanks @Christian_Dahlqvist for now looks like increasing the heap size solved the problem and its working again!! Thanks a ton!

But I would like to have a more permanent solution... will check out the split index api.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 26, 2018, 8:19am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/9 "2018-11-26T08:19:25Z")

</div>

What is your use case? Are you performing a lot of document updates or just adding and deleting?

---

<div class="post-metadata">

**Author:** ![pradla](https://avatars.discourse-cdn.com/v4/letter/p/cab0a1/32.png) [@pradla](https://discuss.elastic.co/u/pradla)\
**Post date:** [November 26, 2018, 8:31am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/10 "2018-11-26T08:31:20Z")

</div>

I am reindexing from remote and copying about 1.5 B tweets from elasticsearch 1.4.4 to elasticsearch 6.5

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 24, 2018, 8:31am UTC](https://discuss.elastic.co/t/facing-error-when-trying-to-reindex-from-remote/158116/11 "2018-12-24T08:31:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.