Elasticsearch bulk insert is too slow on AWS Elasticsearch Service

Elastic Stack Elasticsearch
1

I setup ES 6.0 Service on AWS using m3.xlarge.elasticsearch, i followed Network Limits (http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html).
Bulk insert on my local network is too fast (12 min for 4 millions of document) but too slow on AWS (4 hours for same documents).
I set the replicas to 0 ,refresh_intervall "-1" and other things that said for performance improvement, but no any luck.

Can anyone help to find the solution?
There is same issue in Elasticsearch on aws ec2 index very slow but it closed without any succeed message!!!

2

Have you asked AWS? Do you have any monitoring on the nodes?

3

How many nodes do you have in your cluster? What is your average document size? What type of data are you indexing?

4

Not asked from AWS. No, how can monitor nodes on AWS?

5

I tested with 1 node and 3 nodes. I also tested with different size 1MB, 5MB, 50MB, 90MB, but no success.
The data are person text information.

6

What is the latency to AWS from the machine you are loading from? Are the sizes you specified the average document size you used?

How are you loading data? What is the size of your bulk requests? How many parallell indexing threads/processes are you using?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.