Performance slowdown after upgrading from 0.90 to 1.1.1

We are testing out release 1.1.1 and during our indexing performance
testing, we seemed to get significantly slower throughput, the
document/second rate is about 30% slower. We used the exact same yml file
and startup settings. The code is also identical except for the breaking
changes in the java client api (in this case minor naming changes) and
different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4
shards and no replicas/refreshing. We are indexing documents that are about
2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the
default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:

We are testing out release 1.1.1 and during our indexing performance
testing, we seemed to get significantly slower throughput, the
document/second rate is about 30% slower. We used the exact same yml file
and startup settings. The code is also identical except for the breaking
changes in the java client api (in this case minor naming changes) and
different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4
shards and no replicas/refreshing. We are indexing documents that are about
2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the
default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I tried various counts, after a certain point they didn't make much difference. Also I am not necessarily concerned with improving performance as much as figuring out why I got a slowdown with the exact same settings in 1.1.1. I want to be sure we didn't miss some configuration somewhere or some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:
We are testing out release 1.1.1 and during our indexing performance testing, we seemed to get significantly slower throughput, the document/second rate is about 30% slower. We used the exact same yml file and startup settings. The code is also identical except for the breaking changes in the java client api (in this case minor naming changes) and different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4 shards and no replicas/refreshing. We are indexing documents that are about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:

I tried various counts, after a certain point they didn’t make much
difference. Also I am not necessarily concerned with improving performance
as much as figuring out why I got a slowdown with the exact same settings
in 1.1.1. I want to be sure we didn’t miss some configuration somewhere or
some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:

We are testing out release 1.1.1 and during our indexing performance
testing, we seemed to get significantly slower throughput, the
document/second rate is about 30% slower. We used the exact same yml file
and startup settings. The code is also identical except for the breaking
changes in the java client api (in this case minor naming changes) and
different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4
shards and no replicas/refreshing. We are indexing documents that are about
2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the
default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

*/etc/elasticsearch/elasticsearch.yml : *

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

1.7.0_17

On May 8, 2014, at 6:52 PM, Mark Walkom markw@campaignmonitor.com wrote:

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:
I tried various counts, after a certain point they didn't make much difference. Also I am not necessarily concerned with improving performance as much as figuring out why I got a slowdown with the exact same settings in 1.1.1. I want to be sure we didn't miss some configuration somewhere or some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:
We are testing out release 1.1.1 and during our indexing performance testing, we seemed to get significantly slower throughput, the document/second rate is about 30% slower. We used the exact same yml file and startup settings. The code is also identical except for the breaking changes in the java client api (in this case minor naming changes) and different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4 shards and no replicas/refreshing. We are indexing documents that are about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true
network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]
indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed
threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

OpenJDK or Oracle?

(The current recommended version is 1.7.0_55.)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:55, Kireet kireetreddy@gmail.com wrote:

1.7.0_17

On May 8, 2014, at 6:52 PM, Mark Walkom markw@campaignmonitor.com wrote:

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:

I tried various counts, after a certain point they didn’t make much
difference. Also I am not necessarily concerned with improving performance
as much as figuring out why I got a slowdown with the exact same settings
in 1.1.1. I want to be sure we didn’t miss some configuration somewhere or
some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com
wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:

We are testing out release 1.1.1 and during our indexing performance
testing, we seemed to get significantly slower throughput, the
document/second rate is about 30% slower. We used the exact same yml file
and startup settings. The code is also identical except for the breaking
changes in the java client api (in this case minor naming changes) and
different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with
4 shards and no replicas/refreshing. We are indexing documents that are
about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from
the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

*/etc/elasticsearch/elasticsearch.yml : *

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZoSjN25eaUYc4xr%2BOeBLq4qriN_Q%2BCK9BWhp6H-3qTZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

oracle. Yes I saw that, but I didn't see anything in the release notes mentioning a performance difference in the 2 jdks. I think there was something about some bugfixes. Unless there was something specific, I wouldn't think a minor jdk version would cause a huge performance drop, though I guess stranger things have happened.

On May 8, 2014, at 6:57 PM, Mark Walkom markw@campaignmonitor.com wrote:

OpenJDK or Oracle?

(The current recommended version is 1.7.0_55.)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:55, Kireet kireetreddy@gmail.com wrote:
1.7.0_17

On May 8, 2014, at 6:52 PM, Mark Walkom markw@campaignmonitor.com wrote:

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:
I tried various counts, after a certain point they didn't make much difference. Also I am not necessarily concerned with improving performance as much as figuring out why I got a slowdown with the exact same settings in 1.1.1. I want to be sure we didn't miss some configuration somewhere or some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:
We are testing out release 1.1.1 and during our indexing performance testing, we seemed to get significantly slower throughput, the document/second rate is about 30% slower. We used the exact same yml file and startup settings. The code is also identical except for the breaking changes in the java client api (in this case minor naming changes) and different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4 shards and no replicas/refreshing. We are indexing documents that are about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true
network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]
indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed
threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZoSjN25eaUYc4xr%2BOeBLq4qriN_Q%2BCK9BWhp6H-3qTZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2F401176-E005-4946-BDE3-0395943C798E%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

Which version of 0.90 did you upgrade from? Version 0.90.1 added store
level throttle which can be a bottleneck if you have fast disks:

That is the only change I can think of that would be related to an upgrade.

--
Ivan

On Thu, May 8, 2014 at 4:05 PM, Kireet kireetreddy@gmail.com wrote:

oracle. Yes I saw that, but I didn’t see anything in the release notes
mentioning a performance difference in the 2 jdks. I think there was
something about some bugfixes. Unless there was something specific, I
wouldn’t think a minor jdk version would cause a huge performance drop,
though I guess stranger things have happened.

On May 8, 2014, at 6:57 PM, Mark Walkom markw@campaignmonitor.com wrote:

OpenJDK or Oracle?

(The current recommended version is 1.7.0_55.)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:55, Kireet kireetreddy@gmail.com wrote:

1.7.0_17

On May 8, 2014, at 6:52 PM, Mark Walkom markw@campaignmonitor.com
wrote:

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:

I tried various counts, after a certain point they didn’t make much
difference. Also I am not necessarily concerned with improving performance
as much as figuring out why I got a slowdown with the exact same settings
in 1.1.1. I want to be sure we didn’t miss some configuration somewhere or
some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com
wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:

We are testing out release 1.1.1 and during our indexing performance
testing, we seemed to get significantly slower throughput, the
document/second rate is about 30% slower. We used the exact same yml file
and startup settings. The code is also identical except for the breaking
changes in the java client api (in this case minor naming changes) and
different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with
4 shards and no replicas/refreshing. We are indexing documents that are
about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from
the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

*/etc/elasticsearch/elasticsearch.yml : *

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true

network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]

indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed

threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZoSjN25eaUYc4xr%2BOeBLq4qriN_Q%2BCK9BWhp6H-3qTZA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624ZoSjN25eaUYc4xr%2BOeBLq4qriN_Q%2BCK9BWhp6H-3qTZA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2F401176-E005-4946-BDE3-0395943C798E%40gmail.comhttps://groups.google.com/d/msgid/elasticsearch/2F401176-E005-4946-BDE3-0395943C798E%40gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDMtjuQ6SCkwpM9pz3sV0p4DuM8tXps34ycuh4u_Et6xg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

We upgraded from 0.90.11, but we didn't set this option. Today we were able to reproduce the slowdown on a single node cluster. We also use some analysis plugins, especially the ICU plugin. We are going to try to check performance if we don't use those plugins to see if that's causing the issue.

On May 9, 2014, at 2:20 PM, Ivan Brusic ivan@brusic.com wrote:

Which version of 0.90 did you upgrade from? Version 0.90.1 added store level throttle which can be a bottleneck if you have fast disks: Elasticsearch Platform — Find real-time answers at scale | Elastic

That is the only change I can think of that would be related to an upgrade.

--
Ivan

On Thu, May 8, 2014 at 4:05 PM, Kireet kireetreddy@gmail.com wrote:
oracle. Yes I saw that, but I didn't see anything in the release notes mentioning a performance difference in the 2 jdks. I think there was something about some bugfixes. Unless there was something specific, I wouldn't think a minor jdk version would cause a huge performance drop, though I guess stranger things have happened.

On May 8, 2014, at 6:57 PM, Mark Walkom markw@campaignmonitor.com wrote:

OpenJDK or Oracle?

(The current recommended version is 1.7.0_55.)

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:55, Kireet kireetreddy@gmail.com wrote:
1.7.0_17

On May 8, 2014, at 6:52 PM, Mark Walkom markw@campaignmonitor.com wrote:

What java version?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 08:39, Kireet kireetreddy@gmail.com wrote:
I tried various counts, after a certain point they didn't make much difference. Also I am not necessarily concerned with improving performance as much as figuring out why I got a slowdown with the exact same settings in 1.1.1. I want to be sure we didn't miss some configuration somewhere or some other issue. Thanks!

On May 8, 2014, at 6:32 PM, Mark Walkom markw@campaignmonitor.com wrote:

Can you yry increasing your bulk count to 1000, or more?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 05:01, slushi kireetreddy@gmail.com wrote:
We are testing out release 1.1.1 and during our indexing performance testing, we seemed to get significantly slower throughput, the document/second rate is about 30% slower. We used the exact same yml file and startup settings. The code is also identical except for the breaking changes in the java client api (in this case minor naming changes) and different elasticsearch/lucene jars.

We have a 4 node test cluster. The test basically creates an index with 4 shards and no replicas/refreshing. We are indexing documents that are about 2KB each using the bulk api (500 documents per request).

Below is some environment info and some settings that we changed from the default.

MemTotal: 99016988 kB

ES_HEAP_SIZE=24g

MAX_OPEN_FILES=65535

/etc/elasticsearch/elasticsearch.yml :

cluster.name: estest

node.name: "es1"

node.rack: rack2

bootstrap.mlockall: true
network.host: 1.2.3.4

gateway.recover_after_nodes: 3

gateway.expected_nodes: 3

discovery.zen.minimum_master_nodes: 1

discovery.zen.ping.multicast.enabled: false

discovery.zen.ping.unicast.hosts: ["es1", "es2", "es3", "es4"]
indices.memory.index_buffer_size: 50%

index.translog.flush_threshold_ops: 50000

threadpool.search.type: fixed

threadpool.search.size: 20

threadpool.search.queue_size: 100

threadpool.index.type: fixed
threadpool.index.size: 60

threadpool.index.queue_size: 200

threadpool.bulk.type: fixed

threadpool.bulk.size: 50

threadpool.bulk.queue_size: 1000

cluster.routing.allocation.awareness.attributes: rack

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/edea9e8f-e2f3-424e-b381-3d6dc4b96979%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YYMESG9zYtb9shwEHU1Pkt9Dzn9n%3D6TMegRcx4pyZSkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E64BA391-4124-4CAC-BC70-360F91B549BF%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bJ2JaF9p-P5tkvzCARBp4N7o%2BOJJPs1PO8HXo1q%2Bqy2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/088AABB9-5201-4112-BD5A-449EF5852439%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZoSjN25eaUYc4xr%2BOeBLq4qriN_Q%2BCK9BWhp6H-3qTZA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2F401176-E005-4946-BDE3-0395943C798E%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/qydm3PG4Jxw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDMtjuQ6SCkwpM9pz3sV0p4DuM8tXps34ycuh4u_Et6xg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/32EF6AF1-7584-4CB4-B120-404DEFD19FFC%40gmail.com.
For more options, visit https://groups.google.com/d/optout.