Bulk API to Elastic Cloud write: connection reset by peer

[werowe (Walker Rowe)]

I am trying to bulk load to elastic cloud and get connection reset by peer error. Looks like it is getting trying to write back to my server using a 10.* IP address. I'm not supposed to log into the command line shell on the cloud instance I believe but access it from localhost shell.

If I try to run it on my local server it says

413 Request Entity Too Large

413 Request Entity Too Large

---

nginx/1.10.3 (Ubuntu)

curl --user $pwd -H 'Content-Type: application/x-ndjson' -XPOST 'https://58571402f5464923883e7be42a037917.eu-central-1.aws.cloud.es.io:9243/0/_bulk?pretty' --data-binary @drug-event-0004-of-0004.json

Post https://10.11.131.72:18241/0/_bulk?pretty: net/http: HTTP/1.x transport connection broken: write tcp 172.17.0.2:51240->10.11.131.72:18241: write: connection reset by peer

[dadoonet (David Pilato)]

How big is your json file?

[werowe (Walker Rowe)]

637M

If that's too big can I bulk load the zip file?

[werowe (Walker Rowe)]

The cluster is large (see image)

[werowe (Walker Rowe)]

file is there if anyone else using elastic cloud wants to try https://download.open.fda.gov/drug/event/all_other/drug-event-0004-of-0004.json.zip

[dadoonet (David Pilato)]

Http limit is 100mb iirc so yes split into multiple bulks.
I'd do something like 10mb at a time and try to decrease or increase.