Not getting Good Write Performance

Hi All,

I am creating index using elasticsearch API but not getting good write
speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings =
ImmutableSettings.settingsBuilder().put("node.local",true).build();
client = nodeBuilder().client(true).node().client();

    } else {
         Settings settings = ImmutableSettings.settingsBuilder()
                 .build();
         client = new TransportClient(settings).addTransportAddress(new 

InetSocketTransportAddress(
elasticSearchHost, elasticSearchPort));
}
//Record Dumping Part

  •  * Long MobileNumber= new Long(0);
    

System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length);
Long k=null,f=null;
k=new Date().getTime();

            for(Long i=(long) 0;i<10000;i++)
            {
            IndexResponse response2 = 

this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();

          MobileNumber++;  
            }
            f=new Date().getTime();
            System.out.println("Indexing Speed = "+((240)/(f-k))+"  

Time "+ (f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

Use the Bulk API: https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113

Le 28 janv. 2013 à 17:54, Ankit Jain ankitjaincs06@gmail.com a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings = ImmutableSettings.settingsBuilder().put("node.local",true).build();
client = nodeBuilder().client(true).node().client();
} else {
Settings settings = ImmutableSettings.settingsBuilder()
.build();
client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress(
elasticSearchHost, elasticSearchPort));
}
//Record Dumping Part
Long MobileNumber= new Long(0);
System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length);
Long k=null,f=null;
k=new Date().getTime();

            for(Long i=(long) 0;i<10000;i++)
            {
            IndexResponse response2 = this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();             
          MobileNumber++;  
            }
            f=new Date().getTime();
            System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+ (f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--

I had heard at one point that using the Java API per document vs the bulk
APIs there isn't much difference, perhaps that is (or never was the case).

Anyways, for that specific code chunk, you're calling actionGet on every
call, which makes the call synchronous. Instead you should save off the
action futures and then call the actionGet after they have all been
submitted.

I'd be curious to know how that performs compared to the bulk API.

Best Regards,
Paul

On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:

Use the Bulk API:
https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113

Le 28 janv. 2013 à 17:54, Ankit Jain <ankitj...@gmail.com <javascript:>>
a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good write
speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings =
ImmutableSettings.settingsBuilder().put("node.local",true).build();
client = nodeBuilder().client(true).node().client();

    } else {
         Settings settings = ImmutableSettings.settingsBuilder()
                 .build();
         client = new 

TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress(
elasticSearchHost, elasticSearchPort));
}
//Record Dumping Part

  •  * Long MobileNumber= new Long(0);
    

System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length);
Long k=null,f=null;
k=new Date().getTime();

            for(Long i=(long) 0;i<10000;i++)
            {
            IndexResponse response2 = 

this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();

          MobileNumber++;  
            }
            f=new Date().getTime();
            System.out.println("Indexing Speed = "+((240)/(f-k))+"  

Time "+ (f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

There is a difference:

  • writing 100 docs per single doc request: 100 packets to the cluster,
    cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

  • writing 100 doc in a bulk request: 1 packet to the cluster and 1 back
    to the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end
up in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

I had heard at one point that using the Java API per document vs the
bulk APIs there isn't much difference, perhaps that is (or never was
the case).

Anyways, for that specific code chunk, you're calling actionGet on
every call, which makes the call synchronous. Instead you should save
off the action futures and then call the actionGet after they have all
been submitted.

I'd be curious to know how that performs compared to the bulk API.

Best Regards,
Paul

On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:

Use the Bulk API:
https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113
<https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113>






Le 28 janv. 2013 à 17:54, Ankit Jain <ankitj...@gmail.com
<javascript:>> a écrit :
Hi All,

I am creating index using elasticsearch API but not getting good
write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings =
ImmutableSettings.settingsBuilder().put("node.local",true).build();
             client = nodeBuilder().client(true).node().client();
        } else {
             Settings settings = ImmutableSettings.settingsBuilder()
                     .build();
             client = new
TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress(
                     elasticSearchHost, elasticSearchPort));
        }
//Record Dumping Part
**Long MobileNumber= new Long(0);
               
System.out.println(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes().length);
                 Long k=null,f=null;
                 k=new Date().getTime();

                for(Long i=(long) 0;i<10000;i++)
                {
                IndexResponse response2 =
this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{\"mobileNumberUserId\":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();
              MobileNumber++;
                }
                f=new Date().getTime();
System.out.println("Indexing Speed = "+((240)/(f-k))+"  Time   "+
(f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group, send email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I add to Jorg answer that with bulk, Elasticsearch prepares documents for each shard. Then it has only to send the right docs to the right shard, index all, instead of doing this one by one.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 janv. 2013 à 21:32, Jörg Prante joergprante@gmail.com a écrit :

There is a difference:

  • writing 100 docs per single doc request: 100 packets to the cluster, cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

  • writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end up in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

I had heard at one point that using the Java API per document vs the bulk APIs there isn't much difference, perhaps that is (or never was the case).

Anyways, for that specific code chunk, you're calling actionGet on every call, which makes the call synchronous. Instead you should save off the action futures and then call the actionGet after they have all been submitted.

I'd be curious to know how that performs compared to the bulk API.

Best Regards,
Paul

On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:

Use the Bulk API:
https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113
https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113

Le 28 janv. 2013 à 17:54, Ankit Jain <ankitj...@gmail.com
<javascript:>> a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good
write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings =
ImmutableSettings.settingsBuilder().put("node.local",true).build();
client = nodeBuilder().client(true).node().client();
} else {
Settings settings = ImmutableSettings.settingsBuilder()
.build();
client = new
TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress(
elasticSearchHost, elasticSearchPort));
}
//Record Dumping Part
**Long MobileNumber= new Long(0);
System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length);
Long k=null,f=null;
k=new Date().getTime();

               for(Long i=(long) 0;i<10000;i++)
               {
               IndexResponse response2 =

this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();
MobileNumber++;
}
f=new Date().getTime();
System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+
(f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yeah, that's valid point regarding the extra network I/O, it really comes
down to how quickly you want data showing up in search and if it makes
sense to batch items together depending on your use case.

The thread congestion can be mitigated with a different threadpool type for
indexing:
http://www.elasticsearch.org/guide/reference/modules/threadpool.html

Best Regards,
Paul

On Monday, January 28, 2013 1:37:37 PM UTC-7, David Pilato wrote:

I add to Jorg answer that with bulk, Elasticsearch prepares documents for
each shard. Then it has only to send the right docs to the right shard,
index all, instead of doing this one by one.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 28 janv. 2013 à 21:32, Jörg Prante <joerg...@gmail.com <javascript:>>
a écrit :

There is a difference:

  • writing 100 docs per single doc request: 100 packets to the cluster,
    cluster must acknowledge the 100 packets on the wire = 100 roundtrips.

  • writing 100 doc in a bulk request: 1 packet to the cluster and 1 back to
    the client (with reasonable doc size), 1 roundtrip.

So you save huge communication overhead.

Saving actionGet's only creates thread congestion, each will still end up
in a single request and response on the wire.

Jörg

Am 28.01.13 21:23, schrieb ppearcy:

I had heard at one point that using the Java API per document vs the
bulk APIs there isn't much difference, perhaps that is (or never was the
case).

Anyways, for that specific code chunk, you're calling actionGet on every
call, which makes the call synchronous. Instead you should save off the
action futures and then call the actionGet after they have all been
submitted.

I'd be curious to know how that performs compared to the bulk API.

Best Regards,
Paul

On Monday, January 28, 2013 10:05:17 AM UTC-7, David Pilato wrote:

Use the Bulk API:

https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113

<
https://github.com/elasticsearchfr/hands-on/blob/answers/src/test/java/org/elasticsearchfr/handson/ex1/IndexTest.java#L113>

Le 28 janv. 2013 à 17:54, Ankit Jain <ankitj...@gmail.com
<javascript:>> a écrit :

Hi All,

I am creating index using elasticsearch API but not getting good
write speed.

Below is code that I used for creating client and record dump.
//creates Client in our case
if (localMode != null && localMode) {
Settings settings =
ImmutableSettings.settingsBuilder().put("node.local",true).build();
client = nodeBuilder().client(true).node().client();
} else {
Settings settings = ImmutableSettings.settingsBuilder()
.build();
client = new
TransportClient(settings).addTransportAddress(new
InetSocketTransportAddress(
elasticSearchHost, elasticSearchPort));
}
//Record Dumping Part
**Long MobileNumber= new Long(0);

System.out.println(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes().length);

                Long k=null,f=null; 
                k=new Date().getTime(); 

               for(Long i=(long) 0;i<10000;i++) 
               { 
               IndexResponse response2 = 

this.client.prepareIndex("dummymobilenumber4","Number",MobileNumber.toString()).setSource(("{"mobileNumberUserId":"+MobileNumber.toString()+"}").getBytes()).execute().actionGet();

             MobileNumber++; 
               } 
               f=new Date().getTime(); 

System.out.println("Indexing Speed = "+((240)/(f-k))+" Time "+
(f-k));

Guys, required you help to increase the write performance.

Thanks,
Regards,
Ankit Jain

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group, send email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group, send email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group, send email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.