# Not getting Good Write Performance **URL:** \ **Category:** Elasticsearch\ **Created:** [January 28, 2013, 4:54pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527 "2013-01-28T16:54:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ankit\_Jain](https://avatars.discourse-cdn.com/v4/letter/a/96bed5/32.png) [@Ankit\_Jain](https://discuss.elastic.co/u/Ankit_Jain)\ **Post date:** [January 28, 2013, 4:54pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/1 "2013-01-28T16:54:52Z")
Hi All, I am creating index using elasticsearch API but not getting good write speed. Below is code that I used for creating client and record dump. //creates Client in our case if (localMode != null && localMode) { Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build(); client = nodeBuilder().client(true).node().client(); ``` } else { Settings settings = ImmutableSettings.settingsBuilder() .build(); client = new TransportClient(settings).addTransportAddress(new ``` InetSocketTransportAddress( elasticSearchHost, elasticSearchPort)); } //Record Dumping Part - ``` * Long MobileNumber= new Long(0); ``` System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length); Long k=null,f=null; k=new Date().getTime(); ``` for(Long i=(long) 0;i<10000;i++) { IndexResponse response2 = ``` this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); ``` MobileNumber++; } f=new Date().getTime(); System.out.println("Indexing Speed = "+((240)/(f-k))+" ``` Time "+ (f-k)); Guys, required you help to increase the write performance. Thanks, Regards, Ankit Jain ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 28, 2013, 5:05pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/2 "2013-01-28T17:05:17Z")
Use the Bulk API: [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113) Le 28 janv. 2013 à 17:54, Ankit Jain [ankitjaincs06@gmail.com](mailto:ankitjaincs06@gmail.com) a écrit : > Hi All, > > I am creating index using elasticsearch API but not getting good write speed. > > Below is code that I used for creating client and record dump. > //creates Client in our case > if (localMode != null && localMode) { > Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build(); > client = nodeBuilder().client(true).node().client(); > } else { > Settings settings = ImmutableSettings.settingsBuilder() > .build(); > client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress( > elasticSearchHost, elasticSearchPort)); > } > //Record Dumping Part > Long MobileNumber= new Long(0); > System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length); > Long k=null,f=null; > k=new Date().getTime(); > > ``` > for(Long i=(long) 0;i<10000;i++) > { > IndexResponse response2 = this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); > MobileNumber++; > } > f=new Date().getTime(); > System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+ (f-k)); > > ``` > > Guys, required you help to increase the write performance. > > Thanks, > Regards, > Ankit Jain -- ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [January 28, 2013, 8:23pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/3 "2013-01-28T20:23:00Z")
I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case). Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted. I'd be curious to know how that performs compared to the bulk API. Best Regards, Paul On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote: > Use the Bulk API: > [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113) > > Le 28 janv. 2013 à 17:54, Ankit Jain \<[ankitj...@gmail.com](mailto:ankitj...@gmail.com) \\> > a écrit : > > Hi All, > > I am creating index using elasticsearch API but not getting good write > speed. > > Below is code that I used for creating client and record dump. > //creates Client in our case > if (localMode != null && localMode) { > Settings settings = > ImmutableSettings.settingsBuilder().put("node.local",true).build(); > client = nodeBuilder().client(true).node().client(); > > ``` > } else { > Settings settings = ImmutableSettings.settingsBuilder() > .build(); > client = new > > ``` > > TransportClient(settings).addTransportAddress(new > InetSocketTransportAddress( > elasticSearchHost, elasticSearchPort)); > } > //Record Dumping Part > > - > > ``` > * Long MobileNumber= new Long(0); > > ``` > > System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length); > Long k=null,f=null; > k=new Date().getTime(); > > ``` > for(Long i=(long) 0;i<10000;i++) > { > IndexResponse response2 = > > ``` > > this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); > > ``` > MobileNumber++; > } > f=new Date().getTime(); > System.out.println("Indexing Speed = "+((240)/(f-k))+" > > ``` > > Time "+ (f-k)); > > Guys, required you help to increase the write performance. > > Thanks, > Regards, > Ankit Jain -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 28, 2013, 8:32pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/4 "2013-01-28T20:32:01Z")
There is a difference: - writing 100 docs per single doc request: 100 packets to the cluster, cluster must acknowledge the 100 packets on the wire = 100 roundtrips. - writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to the client (with reasonable doc size), 1 roundtrip. So you save huge communication overhead. Saving actionGet's only creates thread congestion, each will still end up in a single request and response on the wire. Jörg Am 28.01.13 21:23, schrieb ppearcy: > I had heard at one point that using the Java API per document vs the > bulk APIs there isn't much difference, perhaps that is (or never was > the case). > > Anyways, for that specific code chunk, you're calling actionGet on > every call, which makes the call synchronous. Instead you should save > off the action futures and then call the actionGet after they have all > been submitted. > > I'd be curious to know how that performs compared to the bulk API. > > Best Regards, > Paul > > On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote: > > ``` > Use the Bulk API: > https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113 > > > Le 28 janv. 2013 à 17:54, Ankit Jain > a écrit : > > ``` > > > ``` > > Hi All, > > > > I am creating index using elasticsearch API but not getting good > > write speed. > > > > Below is code that I used for creating client and record dump. > > //creates Client in our case > > if (localMode != null && localMode) { > > Settings settings = > > ImmutableSettings.settingsBuilder().put("node.local",true).build(); > > client = nodeBuilder().client(true).node().client(); > > } else { > > Settings settings = ImmutableSettings.settingsBuilder() > > .build(); > > client = new > > TransportClient(settings).addTransportAddress(new > > InetSocketTransportAddress( > > elasticSearchHost, elasticSearchPort)); > > } > > //Record Dumping Part > > **Long MobileNumber= new Long(0); > > > > System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length); > > Long k=null,f=null; > > k=new Date().getTime(); > > > > for(Long i=(long) 0;i<10000;i++) > > { > > IndexResponse response2 = > > this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); > > MobileNumber++; > > } > > f=new Date().getTime(); > > System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+ > > (f-k)); > > > > Guys, required you help to increase the write performance. > > > > Thanks, > > Regards, > > Ankit Jain > > > > ``` > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group, send email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 28, 2013, 8:37pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/5 "2013-01-28T20:37:37Z")
I add to Jorg answer that with bulk, Elasticsearch prepares documents for each shard. Then it has only to send the right docs to the right shard, index all, instead of doing this one by one. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 28 janv. 2013 à 21:32, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) a écrit : There is a difference: - writing 100 docs per single doc request: 100 packets to the cluster, cluster must acknowledge the 100 packets on the wire = 100 roundtrips. - writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to the client (with reasonable doc size), 1 roundtrip. So you save huge communication overhead. Saving actionGet's only creates thread congestion, each will still end up in a single request and response on the wire. Jörg Am 28.01.13 21:23, schrieb ppearcy: > I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case). > > Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted. > > I'd be curious to know how that performs compared to the bulk API. > > Best Regards, > Paul > > On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote: > > Use the Bulk API: > [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113) > [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113) > > Le 28 janv. 2013 à 17:54, Ankit Jain \<[ankitj...@gmail.com](mailto:ankitj...@gmail.com) > \\> a écrit : > > > Hi All, > > > > I am creating index using elasticsearch API but not getting good > > write speed. > > > > Below is code that I used for creating client and record dump. > > //creates Client in our case > > if (localMode != null && localMode) { > > Settings settings = > > ImmutableSettings.settingsBuilder().put("node.local",true).build(); > > client = nodeBuilder().client(true).node().client(); > > } else { > > Settings settings = ImmutableSettings.settingsBuilder() > > .build(); > > client = new > > TransportClient(settings).addTransportAddress(new > > InetSocketTransportAddress( > > elasticSearchHost, elasticSearchPort)); > > } > > //Record Dumping Part > > \*\*Long MobileNumber= new Long(0); > > System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length); > > Long k=null,f=null; > > k=new Date().getTime(); > > > > ``` > > for(Long i=(long) 0;i<10000;i++) > > { > > IndexResponse response2 = > > > > ``` > > > > this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); > > MobileNumber++; > > } > > f=new Date().getTime(); > > System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+ > > (f-k)); > > > > Guys, required you help to increase the write performance. > > > > Thanks, > > Regards, > > Ankit Jain > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [January 28, 2013, 8:44pm UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/6 "2013-01-28T20:44:54Z")
Yeah, that's valid point regarding the extra network I/O, it really comes down to how quickly you want data showing up in search and if it makes sense to batch items together depending on your use case. The thread congestion can be mitigated with a different threadpool type for indexing: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Best Regards, Paul On Monday, January 28, 2013 1:37:37 PM UTC-7, David Pilato wrote: > I add to Jorg answer that with bulk, Elasticsearch prepares documents for > each shard. Then it has only to send the right docs to the right shard, > index all, instead of doing this one by one. > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 28 janv. 2013 à 21:32, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com) \\> > a écrit : > > There is a difference: > > - writing 100 docs per single doc request: 100 packets to the cluster, > cluster must acknowledge the 100 packets on the wire = 100 roundtrips. > > - writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to > the client (with reasonable doc size), 1 roundtrip. > > So you save huge communication overhead. > > Saving actionGet's only creates thread congestion, each will still end up > in a single request and response on the wire. > > Jörg > > Am 28.01.13 21:23, schrieb ppearcy: > > > I had heard at one point that using the Java API per document vs the > > bulk APIs there isn't much difference, perhaps that is (or never was the > > case). > > > > Anyways, for that specific code chunk, you're calling actionGet on every > > call, which makes the call synchronous. Instead you should save off the > > action futures and then call the actionGet after they have all been > > submitted. > > > > I'd be curious to know how that performs compared to the bulk API. > > > > Best Regards, > > Paul > > > > On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote: > > > > Use the Bulk API: > > [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113) > > > \< > > [https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113](https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113)\> > > > Le 28 janv. 2013 à 17:54, Ankit Jain \<[ankitj...@gmail.com](mailto:ankitj...@gmail.com) > > \\> a écrit : > > > > > Hi All, > > > > > > I am creating index using elasticsearch API but not getting good > > > write speed. > > > > > > Below is code that I used for creating client and record dump. > > > //creates Client in our case > > > if (localMode != null && localMode) { > > > Settings settings = > > > ImmutableSettings.settingsBuilder().put("node.local",true).build(); > > > client = nodeBuilder().client(true).node().client(); > > > } else { > > > Settings settings = ImmutableSettings.settingsBuilder() > > > .build(); > > > client = new > > > TransportClient(settings).addTransportAddress(new > > > InetSocketTransportAddress( > > > elasticSearchHost, elasticSearchPort)); > > > } > > > //Record Dumping Part > > > \*\*Long MobileNumber= new Long(0); > > System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length); > > > > ``` > > > Long k=null,f=null; > > > k=new Date().getTime(); > > > > > > for(Long i=(long) 0;i<10000;i++) > > > { > > > IndexResponse response2 = > > > > > > ``` > > this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet(); > > > > ``` > > > MobileNumber++; > > > } > > > f=new Date().getTime(); > > > > > > ``` > > > > > > System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+ > > > (f-k)); > > > > > > Guys, required you help to increase the write performance. > > > > > > Thanks, > > > Regards, > > > Ankit Jain > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group, send email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group, send email to > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group, send email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:54am UTC](https://discuss.elastic.co/t/not-getting-good-write-performance/10527/7 "2017-07-06T02:54:18Z")