# Bulk indexing with EC2 cluster? **URL:** \ **Category:** Elasticsearch\ **Created:** [March 25, 2014, 5:52pm UTC](https://discuss.elastic.co/t/bulk-indexing-with-ec2-cluster/16593 "2014-03-25T17:52:57Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![IronMan2014](https://avatars.discourse-cdn.com/v4/letter/i/94ad74/32.png) [@IronMan2014](https://discuss.elastic.co/u/IronMan2014)\ **Post date:** [March 25, 2014, 5:52pm UTC](https://discuss.elastic.co/t/bulk-indexing-with-ec2-cluster/16593/1 "2014-03-25T17:52:57Z")
I am having some issues and I would like some feedback: #1 - I run a test with 250 MB worth of documents against my local machine which is an i7, it takes total of 130 secs to index. I run it against a cluster of 2 i2x4 large EC2 instances, much more powerful than my local machine, yet it takes about 200 secs for the same test. #2, When I index against local machine, it shows 1500 docs indexed total, however on the 2 instances, I see 1150 docs, why is it different. #3, Aside from the above, a separate test, if I run smaller # of docs say 500, The bulk gets called but never executes, the bulk and index.close() exit without the Bulk.execute, I look at the index, it is empty, no docs were actually indexed, but against my local machine this doesn't happen. Some settings: BulkSize: 1000 docs & 5 MB Settings settings = ImmutableSettings.settingsBuilder() ``` .put("client.transport.sniff", true) .put("refresh_interval", "-1") .put("number_of_shards", 1) .put("number_of_replicas", "0") .put("cluster.name", this.CLUSTER_NAME) .build(); ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e55dd715-43dd-45a8-ad50-d543792f0481%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e55dd715-43dd-45a8-ad50-d543792f0481%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 26, 2014, 8:06pm UTC](https://discuss.elastic.co/t/bulk-indexing-with-ec2-cluster/16593/2 "2014-03-26T20:06:33Z")
1. There is network latency going up to EC2. 🙂 2/3. Not sure, can you show your bulk code? Or did you check the logs on your EC2 instances to see if there were any errors? On Tuesday, March 25, 2014 1:52:57 PM UTC-4, IronMan2014 wrote: > I am having some issues and I would like some feedback: > > #1 - I run a test with 250 MB worth of documents against my local machine > which is an i7, it takes total of 130 secs to index. I run it against a > cluster of 2 i2x4 large EC2 instances, much more powerful than my local > machine, yet it takes about 200 secs for the same test. > > #2, When I index against local machine, it shows 1500 docs indexed total, > however on the 2 instances, I see 1150 docs, why is it different. > > #3, Aside from the above, a separate test, if I run smaller # of docs say > 500, The bulk gets called but never executes, the bulk and index.close() > exit without the Bulk.execute, I look at the index, it is empty, no docs > were actually indexed, but against my local machine this doesn't happen. > > Some settings: > > BulkSize: 1000 docs & 5 MB > > Settings settings = ImmutableSettings.settingsBuilder() > > ``` > .put("client.transport.sniff", true > > ``` > > ) > > ``` > .put("refresh_interval", "-1") > > .put("number_of_shards", 1) > > .put("number_of_replicas", "0") > > .put("cluster.name", this.CLUSTER_NAME) > > .build(); > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1a89bbdd-8db3-4499-abc2-ea841a865b56%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1a89bbdd-8db3-4499-abc2-ea841a865b56%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![IronMan2014](https://avatars.discourse-cdn.com/v4/letter/i/94ad74/32.png) [@IronMan2014](https://discuss.elastic.co/u/IronMan2014)\ **Post date:** [March 26, 2014, 8:15pm UTC](https://discuss.elastic.co/t/bulk-indexing-with-ec2-cluster/16593/3 "2014-03-26T20:15:22Z")
Thanks, I was thinking the same about network latency, so I am going to try to run right from the instance itself. I will check on other issues and update my post. On Wednesday, March 26, 2014 4:06:33 PM UTC-4, Binh Ly wrote: > 1. There is network latency going up to EC2. 🙂 > > 2/3. Not sure, can you show your bulk code? Or did you check the logs on > your EC2 instances to see if there were any errors? > > On Tuesday, March 25, 2014 1:52:57 PM UTC-4, IronMan2014 wrote: > > > I am having some issues and I would like some feedback: > > > > #1 - I run a test with 250 MB worth of documents against my local machine > > which is an i7, it takes total of 130 secs to index. I run it against a > > cluster of 2 i2x4 large EC2 instances, much more powerful than my local > > machine, yet it takes about 200 secs for the same test. > > > > #2, When I index against local machine, it shows 1500 docs indexed total, > > however on the 2 instances, I see 1150 docs, why is it different. > > > > #3, Aside from the above, a separate test, if I run smaller # of docs say > > 500, The bulk gets called but never executes, the bulk and index.close() > > exit without the Bulk.execute, I look at the index, it is empty, no docs > > were actually indexed, but against my local machine this doesn't happen. > > > > Some settings: > > > > BulkSize: 1000 docs & 5 MB > > > > Settings settings = ImmutableSettings.settingsBuilder() > > > > ``` > > .put("client.transport.sniff", > > > > ``` > > > > true) > > > > ``` > > .put("refresh_interval", "-1") > > > > .put("number_of_shards", 1) > > > > .put("number_of_replicas", "0") > > > > .put("cluster.name", this.CLUSTER_NAME > > > > ``` > > > > ) > > > > ``` > > .build(); > > > > ``` -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7eddc0ed-3c72-446f-882f-9fbbe9dce9f1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7eddc0ed-3c72-446f-882f-9fbbe9dce9f1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:40am UTC](https://discuss.elastic.co/t/bulk-indexing-with-ec2-cluster/16593/4 "2017-07-06T01:40:21Z")