Unable to connect 502 during a bulk document upload

[lukelindner01 (Luke Lindner)]

Heya. I'm new with Elastic Search and AppSearch but we have setup an engine and am attempting to bulk upload a 100 documents per request and up to 1000 requests, throttled to 10 at a time via a semaphore (c#). I keep on getting the below http exception shortly after initiating the bulk upload (the first couple succeed without issue):

An existing connection was forcibly closed by the remote host.

When I try and view my appsearch engine though, I see that it is unavailable

After some time, it resolves itself, but becomes unavailable again if I reattempt the bulk uploads.

I am assuming my document uploads are crashing our instance/engine. How do I resolve this?

[ross.bell]

Hey @lukelindner01,

You'll likely want to do one or both of:

• Reduce your concurrent indexing request count
• Increase Enterprise Search and Elasticsearch resource capacity (Elasticsearch matters more after a point)

I'd recommend, starting at 1, increasing concurrent indexing request count to understand when indexing requests begin to affect performance. Then you can increase cluster capacity as needed to achieve your desired indexing throughput.

I hope that helps!

Ross

[lukelindner01 (Luke Lindner)]

@ross.bell

concurrent indexing request

Thanks @ross.bell. When you mention the concurrent indexing request count, are you refering to the amount of concurrent document uploads we are attempting at a time?

[ross.bell]

Yep, you mention you're issuing 10 requests in parallel. You'll want to play around with how many parallel requests your cluster size supports. It sounds like it's currently too small to handle 10.