Increase query performance by adding more machines, shouldn't it be linear to # of machines?

Elastic Stack Elasticsearch
1

Hi all,

I'm testing percolator performance, 50k/s is required condition with 3~4k
rules.

now I only have 1 simple rule, and 5 es vms with 1 shard and 4 replicas.

and using Java transport client like below

new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new
InetSocketTransportAddress[transportAddressList.size()]));

when I added just 1 address to transport client, percolator perf is about
10k/s

and when I added all 5 of vms to it, perf is about 15k/s

so it increases by about 1.5times only even though I added 4 more vm
addresses.

Is it supposed to be like this?

What I was thinking is, if it runs in for example round-robin fashion,
there should be about 5 times performance gain.

Could you comment anything on this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

2

Are you using same physical machine for all your VMs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 juil. 2014 à 07:09, Seungjin Lee sweetest0409@gmail.com a écrit :

Hi all,

I'm testing percolator performance, 50k/s is required condition with 3~4k rules.

now I only have 1 simple rule, and 5 es vms with 1 shard and 4 replicas.

and using Java transport client like below

new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new InetSocketTransportAddress[transportAddressList.size()]));

when I added just 1 address to transport client, percolator perf is about 10k/s

and when I added all 5 of vms to it, perf is about 15k/s

so it increases by about 1.5times only even though I added 4 more vm addresses.

Is it supposed to be like this?

What I was thinking is, if it runs in for example round-robin fashion, there should be about 5 times performance gain.

Could you comment anything on this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Also, what's the hardware configuration of the devices? RAM? CPU? Disk? OS?

How much data is in your ES instance(s) already? and are you sure you're
not hitting a limit on something like client side networking?

Thanks,
Patrick

Patrick

patrick eefy net

On Tue, Jul 1, 2014 at 11:55 PM, David Pilato david@pilato.fr wrote:

Are you using same physical machine for all your VMs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 juil. 2014 à 07:09, Seungjin Lee sweetest0409@gmail.com a écrit :

Hi all,

I'm testing percolator performance, 50k/s is required condition with 3~4k
rules.

now I only have 1 simple rule, and 5 es vms with 1 shard and 4 replicas.

and using Java transport client like below

new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new
InetSocketTransportAddress[transportAddressList.size()]));

when I added just 1 address to transport client, percolator perf is about
10k/s

and when I added all 5 of vms to it, perf is about 15k/s

so it increases by about 1.5times only even though I added 4 more vm
addresses.

Is it supposed to be like this?

What I was thinking is, if it runs in for example round-robin fashion,
there should be about 5 times performance gain.

Could you comment anything on this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL%2BHSrOczux-gdHPNDRJcPU%3D4CNEh2uKb13NE-%2BGPP3Kw_NjEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

yes, all same machines on which only ES with same configuration is running

2014-07-02 14:55 GMT+09:00 David Pilato david@pilato.fr:

Are you using same physical machine for all your VMs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 juil. 2014 à 07:09, Seungjin Lee sweetest0409@gmail.com a écrit :

Hi all,

I'm testing percolator performance, 50k/s is required condition with 3~4k
rules.

now I only have 1 simple rule, and 5 es vms with 1 shard and 4 replicas.

and using Java transport client like below

new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new
InetSocketTransportAddress[transportAddressList.size()]));

when I added just 1 address to transport client, percolator perf is about
10k/s

and when I added all 5 of vms to it, perf is about 15k/s

so it increases by about 1.5times only even though I added 4 more vm
addresses.

Is it supposed to be like this?

What I was thinking is, if it runs in for example round-robin fashion,
there should be about 5 times performance gain.

Could you comment anything on this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr
https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U42vJaO0oqfw4DjWP6KK%3DpqtnGEBFGKHOr7Ky7nmbXV5HA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Sorry. I meant on how many physical bare metal machines your 5 VMs are running?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 juil. 2014 à 07:59, Seungjin Lee sweetest0409@gmail.com a écrit :

yes, all same machines on which only ES with same configuration is running

2014-07-02 14:55 GMT+09:00 David Pilato david@pilato.fr:

Are you using same physical machine for all your VMs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 juil. 2014 à 07:09, Seungjin Lee sweetest0409@gmail.com a écrit :

Hi all,

I'm testing percolator performance, 50k/s is required condition with 3~4k rules.

now I only have 1 simple rule, and 5 es vms with 1 shard and 4 replicas.

and using Java transport client like below

new TransportClient(settings)
.addTransportAddresses(transportAddressList.toArray(new InetSocketTransportAddress[transportAddressList.size()]));

when I added just 1 address to transport client, percolator perf is about 10k/s

and when I added all 5 of vms to it, perf is about 15k/s

so it increases by about 1.5times only even though I added 4 more vm addresses.

Is it supposed to be like this?

What I was thinking is, if it runs in for example round-robin fashion, there should be about 5 times performance gain.

Could you comment anything on this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U412ecCDh0GONWJZy6S9zjY%2By8xvoVsPrt1849csfs-zUA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BAEDE7D4-320E-4123-A783-823FF1EC26ED%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL3_U42vJaO0oqfw4DjWP6KK%3DpqtnGEBFGKHOr7Ky7nmbXV5HA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5BFE5974-6B7B-465A-A587-AEBDF6C27D23%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.