0.90.2 performance problem with 5w msg/sec

Elastic Stack Elasticsearch
1

hey guys,

I'm using elasticsearch 0.90.2 with 4 servers in a cluster.

My parameters are below:

shards: 16
replicas: 0
bootstrap.mlockall: true

index.merge.scheduler.max_thread_count: 15

threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

index.merge.policy.merge_factor: 1000
cache.memory.small_buffer_size: 5mb
cache.memory.large_buffer_size: 50mb

JVM is "-Xmx12g -Xms12g -Xmn6g"
*
*
---------------------------------------------------------------
*
*
For our environment, every morning will met a peak time which hits about 5w
msg/sec and the elasticsearch servers' io are 100%.

My question is: is my servers meet the bottleneck of performance? what
actions can I take to improve performance without enlarge the cluster?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

by 5w you mean 50k?

On Tue, Aug 6, 2013 at 10:41 AM, 姚仁捷 baniu.yao@gmail.com wrote:

hey guys,

I'm using elasticsearch 0.90.2 with 4 servers in a cluster.

My parameters are below:

shards: 16
replicas: 0
bootstrap.mlockall: true

index.merge.scheduler.max_thread_count: 15

threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

index.merge.policy.merge_factor: 1000
cache.memory.small_buffer_size: 5mb
cache.memory.large_buffer_size: 50mb

JVM is "-Xmx12g -Xms12g -Xmn6g"
*
*
---------------------------------------------------------------
*
*
For our environment, every morning will met a peak time which hits about
5w msg/sec and the elasticsearch servers' io are 100%.

My question is: is my servers meet the bottleneck of performance? what
actions can I take to improve performance without enlarge the cluster?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Perry | 彭琪
http://pengqi.me

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

yes，w(an)万 mean 10k in Chinese.

2013/8/7 Perry perrywky@gmail.com

by 5w you mean 50k?

On Tue, Aug 6, 2013 at 10:41 AM, 姚仁捷 baniu.yao@gmail.com wrote:

hey guys,

I'm using elasticsearch 0.90.2 with 4 servers in a cluster.

My parameters are below:

shards: 16
replicas: 0
bootstrap.mlockall: true

index.merge.scheduler.max_thread_count: 15

threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

index.merge.policy.merge_factor: 1000
cache.memory.small_buffer_size: 5mb
cache.memory.large_buffer_size: 50mb

JVM is "-Xmx12g -Xms12g -Xmn6g"
*
*
---------------------------------------------------------------
*
*
For our environment, every morning will met a peak time which hits about
5w msg/sec and the elasticsearch servers' io are 100%.

My question is: is my servers meet the bottleneck of performance? what
actions can I take to improve performance without enlarge the cluster?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Perry | 彭琪
http://pengqi.me

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Can you explain why you select

-Xmn6g
index.merge.policy.merge_factor: 1000
index.merge.scheduler.max_thread_count: 15
threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

Note, with the exceptional huge merge factor 1000, you easily run into
heavy and long lasting I/O situations. With the threadpool enlargement, you
put extra burden on the heap (max. 100 x 10000 docs is quite heavy). My
reommendation is, run default thread pool, a merge factor of 20, and
consider fast disks (SSD).

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

yes, 1w=10k, exactly in Chinese:)

在 2013年8月7日星期三UTC+8上午11时00分06秒，chenlin rao写道：

yes，w(an)万 mean 10k in Chinese.

2013/8/7 Perry <perr...@gmail.com <javascript:>>

by 5w you mean 50k?

On Tue, Aug 6, 2013 at 10:41 AM, 姚仁捷 <bani...@gmail.com <javascript:>>wrote:

hey guys,

I'm using elasticsearch 0.90.2 with 4 servers in a cluster.

My parameters are below:

shards: 16
replicas: 0
bootstrap.mlockall: true

index.merge.scheduler.max_thread_count: 15

threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

index.merge.policy.merge_factor: 1000
cache.memory.small_buffer_size: 5mb
cache.memory.large_buffer_size: 50mb

JVM is "-Xmx12g -Xms12g -Xmn6g"
*
*
---------------------------------------------------------------
*
*
For our environment, every morning will met a peak time which hits about
5w msg/sec and the elasticsearch servers' io are 100%.

My question is: is my servers meet the bottleneck of performance? what
actions can I take to improve performance without enlarge the cluster?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Perry | 彭琪
http://pengqi.me

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

hi jorg,

sorry for that i cannot spell your name correctly in my input method:)

this params is a suggestion from my colleague who is more familiar with
lucene than me.

I've observed the docs is about 200million one day and my tuning goal is up
to 1billion docs one day. I think disk io is the bottle-neck in the cluster
and i'd like to add more server to the cluster.

在 2013年8月7日星期三UTC+8下午3时05分48秒，Jörg Prante写道：

Can you explain why you select

-Xmn6g
index.merge.policy.merge_factor: 1000
index.merge.scheduler.max_thread_count: 15
threadpool:
index:
type: fixed
size: 100
queue_size: 10000
reject_policy: caller

Note, with the exceptional huge merge factor 1000, you easily run into
heavy and long lasting I/O situations. With the threadpool enlargement, you
put extra burden on the heap (max. 100 x 10000 docs is quite heavy). My
reommendation is, run default thread pool, a merge factor of 20, and
consider fast disks (SSD).

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.