I'm trying to migrate and index from elastic search 5.5 to 6.8 using remote reindex. The index is 226 GB in size, has 212m documents, and has 10 primary shards (20 total). I am initiating the reindex using the reindex API with a request using curl.
In my testing on smaller indices, I used a batch size of 1000 with no problem, but I quickly ran into a problem with this index and received the error
entity content is too long [127862838] for the configured buffer limit [104857600]
I found two other posts that state the buffer limit cannot be increased and that the only method to resolve this is to reduce the batch size:
- Elasticsearch-rest-client-5.6.1.jar org.apache.http.ContentTooLongException: entity content is too long [217056451] for the configured buffer limit [104857600]
- Reindex http entity content buffer limit should be configurable
So I reduced the batch size. First to 100, then to 10, and eventually to 1. Each step along the way I encountered the entity content is too long error.
Currently with a batch size of 1, it is indexing at a rate of 15/s (2 GB/hour), so I estimate it will take 110 hours to reindex.
I'm asking again to see if there has been any progress or changes to reindex that I can leverage. Is the buffer limit configurable? Is there some other method that can be used to copy an index from an elasticsearch 5.5 to 6.8 cluster?
Thank you.