Benchmarks (again)


(Robin Clarke) #1

I did some intensive tests last week on a 20-node cluster and had the
following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without any
additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of 2500k
events/minute (41k/second), but the bottleneck was the logstash CPU
load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what point
is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what version of
ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had the
following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without any
additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of 2500k
events/minute (41k/second), but the bottleneck was the logstash CPU
load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Robin Clarke-2) #3

Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw
IOWait, but was also focusing on ingestion rate).
I always had 2 master nodes, and in addition tried the configurations 20,
10 and 5 data nodes.
Running Elasticsearch 1.0.1 (but with Logstash 1.3.3)

-Robin-

On 25 March 2014 06:17, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what version
of ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had the
following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without
any additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of 2500k
events/minute (41k/second), but the bottleneck was the logstash CPU
load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

Java version? Also what OS?

Just as a general note, it's always good to have an uneven number of
masters to ensure you get a majority quorum.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 17:35, Robin Clarke robin@robinclarke.net wrote:

Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw
IOWait, but was also focusing on ingestion rate).
I always had 2 master nodes, and in addition tried the configurations 20,
10 and 5 data nodes.
Running Elasticsearch 1.0.1 (but with Logstash 1.3.3)

-Robin-

On 25 March 2014 06:17, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what version
of ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had the
following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without
any additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of
2500k events/minute (41k/second), but the bottleneck was the logstash
CPU load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Robin Clarke-2) #5

Thanks for the tip with the number of masters!

java version "1.6.0_45" on Debian 3.2.54-2

On 25 March 2014 07:55, Mark Walkom markw@campaignmonitor.com wrote:

Java version? Also what OS?

Just as a general note, it's always good to have an uneven number of
masters to ensure you get a majority quorum.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 17:35, Robin Clarke robin@robinclarke.net wrote:

Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw
IOWait, but was also focusing on ingestion rate).
I always had 2 master nodes, and in addition tried the configurations 20,
10 and 5 data nodes.
Running Elasticsearch 1.0.1 (but with Logstash 1.3.3)

-Robin-

On 25 March 2014 06:17, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what version
of ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had the
following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without
any additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of
2500k events/minute (41k/second), but the bottleneck was the
logstash CPU load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #6

Ouch, why such an old version of java? (And is is Open or Oracle?)
v7 is the minimum for ES and LS.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 18:20, Robin Clarke robin@robinclarke.net wrote:

Thanks for the tip with the number of masters!

java version "1.6.0_45" on Debian 3.2.54-2

On 25 March 2014 07:55, Mark Walkom markw@campaignmonitor.com wrote:

Java version? Also what OS?

Just as a general note, it's always good to have an uneven number of
masters to ensure you get a majority quorum.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 17:35, Robin Clarke robin@robinclarke.net wrote:

Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw
IOWait, but was also focusing on ingestion rate).
I always had 2 master nodes, and in addition tried the configurations
20, 10 and 5 data nodes.
Running Elasticsearch 1.0.1 (but with Logstash 1.3.3)

-Robin-

On 25 March 2014 06:17, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what
version of ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had
the following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format) without
any additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of
2500k events/minute (41k/second), but the bottleneck was the
logstash CPU load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zv19Ww7qYthS9ti1XgNF3E0e2eFQ8dsf0aPLA9Z-Mgrw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Robin Clarke-2) #7

Sorry, correction, we were running ES and LS with java version "1.7.0_51",
I just looked up the default on that system, which (someone says) is
required for some other applications... but yes, we were using the correct
Java version! :slight_smile:

-Robin-

On 25 March 2014 08:41, Mark Walkom markw@campaignmonitor.com wrote:

Ouch, why such an old version of java? (And is is Open or Oracle?)
v7 is the minimum for ES and LS.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 18:20, Robin Clarke robin@robinclarke.net wrote:

Thanks for the tip with the number of masters!

java version "1.6.0_45" on Debian 3.2.54-2

On 25 March 2014 07:55, Mark Walkom markw@campaignmonitor.com wrote:

Java version? Also what OS?

Just as a general note, it's always good to have an uneven number of
masters to ensure you get a majority quorum.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 17:35, Robin Clarke robin@robinclarke.net wrote:

Each node had 8 cores (2.4GHz Xeon), 32GB RAM, SSD disks (I never saw
IOWait, but was also focusing on ingestion rate).
I always had 2 master nodes, and in addition tried the configurations
20, 10 and 5 data nodes.
Running Elasticsearch 1.0.1 (but with Logstash 1.3.3)

-Robin-

On 25 March 2014 06:17, Mark Walkom markw@campaignmonitor.com wrote:

Can you elaborate what your cluster setup is like?

Node specs (disk, RAM, CPU), how many master/data nodes and what
version of ES and java you're running?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 25 March 2014 16:11, Robin Clarke robin13@gmail.com wrote:

I did some intensive tests last week on a 20-node cluster and had
the following insights - I'd be interested if anyone has similar/dissimilar
experience.
The had 20 nodes had 8 cores each, and 32GB memory each. I set up
Elasticsearch to have 15GB of that memory.
The sample events I was using were Apache logs (common format)
without any additional fields (no geoip, useragent etc. plugins).
When running as a 20-node cluster, I got a maximum igestion rate of
2500k events/minute (41k/second), but the bottleneck was the
logstash CPU load... so I reduced to a 10 node cluster...
With the 10 nodes I initially had 1600k/minute (27k) and acheived
1800k/minute (30k/second) by increasing index_refresh_interval to 30s and
index_buffer_size to 20%
Further reducing to 5 nodes, I had 1100k/minute (18k/second).
This brings me to an interesting comparison: at 10 nodes, I have 3k
events/second/node, and with 5 nodes I have 3.66k events/second/node. i.e.
the overhead for doubling the number of nodes from 5 to 10 is about 20%.
Is this to be expected? Just how scalable is Elasticsearch - at what
point is the diminishing return on adding nodes not cost effective?
Is the further logical reduction to 375 events/core/second still
meaningful?

Cheers,
-Robin-

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3353dab5-6241-4b41-8845-6c5f8553d488%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624b8a8ZZbOc81g18XZWNz3qYV3gby%3DCtGrPZ8f%2BwiXXYEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vaf_r-GY6KVMk95R-jip-aMdr2y20-WwNL84jQe0PvD1Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624audypF5E%2BDyZcUJmf%2BtddunbLswKPwEJMyHQ_X%2B2xdEw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACX78vYbV7%2BJ7c7_aSjzh7UL4e3vykgws9pFMevjZetarcEGJA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624Zv19Ww7qYthS9ti1XgNF3E0e2eFQ8dsf0aPLA9Z-Mgrw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624Zv19Ww7qYthS9ti1XgNF3E0e2eFQ8dsf0aPLA9Z-Mgrw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vbgqVXOxpNDTVYN4EF_Yg9C_04MZUg8uNxrewmA%2BBbo5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Steinar Bang) #8

Robin Clarke robin@robinclarke.net:

Thanks for the tip with the number of masters!
java version "1.6.0_45" on Debian 3.2.54-2

Debian 3.2? Are you sure...?
http://www.debian.org/releases/woody/
http://www.debian.org/releases/sarge/

AFAIK there is no 3.2. Debian went from 3.1 "woody" to 4.0 "etch".

(I have a computer still running, that was installed as debian potato in
2001, taken to debian testing, and went to woody as the first stable
release in 2002. Both woody or sarge were fine releases in their time,
but I wouldn't pick either for a new high performance cluster...:slight_smile: )

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/upzc1txpky3j.fsf%40dod.no.
For more options, visit https://groups.google.com/d/optout.


(Robin Clarke-2) #9

I just pasted output from uname there - that's kernel 3.2.54-2 you're
reading there.
I think the Debian release is Wheezy (7.0)

-Robin-

On 26 March 2014 12:17, Steinar Bang sb@dod.no wrote:

Robin Clarke robin@robinclarke.net:

Thanks for the tip with the number of masters!
java version "1.6.0_45" on Debian 3.2.54-2

Debian 3.2? Are you sure...?
http://www.debian.org/releases/woody/
http://www.debian.org/releases/sarge/

AFAIK there is no 3.2. Debian went from 3.1 "woody" to 4.0 "etch".

(I have a computer still running, that was installed as debian potato in
2001, taken to debian testing, and went to woody as the first stable
release in 2002. Both woody or sarge were fine releases in their time,
but I wouldn't pick either for a new high performance cluster...:slight_smile: )

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/upzc1txpky3j.fsf%40dod.no.
For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vYdjQb%2B5oW2Bst_2Ar18bzGQVaCFUe0i_4RgLJnE8cdXw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Steinar Bang) #10

Robin Clarke robin@robinclarke.net:

I just pasted output from uname there - that's kernel 3.2.54-2 you're
reading there.
I think the Debian release is Wheezy (7.0)

OK. The OS version can be found in the /etc/debian_version file.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/upzcwqfhjibk.fsf%40dod.no.
For more options, visit https://groups.google.com/d/optout.


(Robin Clarke-2) #11

In that case, 7.4

-Robin-

On 26 March 2014 12:43, Steinar Bang sb@dod.no wrote:

Robin Clarke robin@robinclarke.net:

I just pasted output from uname there - that's kernel 3.2.54-2 you're
reading there.
I think the Debian release is Wheezy (7.0)

OK. The OS version can be found in the /etc/debian_version file.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/muIKhFkrxFc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/upzcwqfhjibk.fsf%40dod.no.
For more options, visit https://groups.google.com/d/optout.

--
Best winds,
-Robin-
~:)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACX78vbeuTZunCKn_-Uo%3DKMabxFJufA5cZadLD2q7dHJ9YFMeA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #12