I've created a stress test to submit 100 concurrent queries to ES
continuesly. After sending 100 requests, waits 1-2 minutes and send the
next 100 requests.
ES configured on a single node with one shard and no replica.
There is only one single ES connection to submit all the queries for all
the runs. Also documents are indexing constantly (1300 docs/s).
After near half an hour, the average search speed decreases to half. If I
stop the test and start it (connection will be closed and a new one
openned) speed will be fine again.
There is no performance problem with indexing.
So my question is, what could be the reason of search speed decrease.
index configuration:
index.compound_format: false
index.version.created: 900399
index.number_of_shards: 1
index.translog.flush_threshold_period: 60s
index.store.throttle.type: merge
index.cache.filter.max_size: 10
index.cache.field.expire: 1m
index.merge.policy.merge_factor: 30
index.cache.filter.expire: 1m
index.number_of_replicas: 1
index.store.throttle.max_bytes_per_sec: 5mb
index.merge.policy.use_compound_file: false
index.store.compress.stored: true
index.cache.field.type: resident
index.indices.memory.index_buffer_size: 20%
ES version is 0.90.3 and I'm using TransportClient to get connections*.*
Are you sure that this is due to long-living connections? I'm asking
because there are lots of moving parts. For example, it could be that
search became slower because large background merges were running and they
had finished before you started the test again? Otherwise, I have no clue
why a long-living connection would have worse performance than a newly-open
one.
I've created a stress test to submit 100 concurrent queries to ES
continuesly. After sending 100 requests, waits 1-2 minutes and send the
next 100 requests.
ES configured on a single node with one shard and no replica.
There is only one single ES connection to submit all the queries for all
the runs. Also documents are indexing constantly (1300 docs/s).
After near half an hour, the average search speed decreases to half. If I
stop the test and start it (connection will be closed and a new one
openned) speed will be fine again.
There is no performance problem with indexing.
So my question is, what could be the reason of search speed decrease.
index configuration:
index.compound_format: false
index.version.created: 900399
index.number_of_shards: 1
index.translog.flush_threshold_period: 60s
index.store.throttle.type: merge
index.cache.filter.max_size: 10
index.cache.field.expire: 1m
index.merge.policy.merge_factor: 30
index.cache.filter.expire: 1m
index.number_of_replicas: 1
index.store.throttle.max_bytes_per_sec: 5mb
index.merge.policy.use_compound_file: false
index.store.compress.stored: true
index.cache.field.type: resident
index.indices.memory.index_buffer_size: 20%
ES version is 0.90.3 and I'm using TransportClient to get connections*.*
the average of response time for 100 queries at each run are written, the
times are in ms: 1th run average time : 371
2th run average time : 358
3th run average time : 1479
4th run average time : 1321
5th run average time : 1661
6th run average time : 1741
7th run average time : 1768
8th run average time : 1928
9th run average time : 1638
10th run average time : 1396
11th run average time : 1682
12th run average time : 1355
13th run average time : 1599
14th run average time : 1250
15th run average time : 1387
16th run average time : 1364
17th run average time : 1044
18th run average time : 1313
19th run average time : 1726
20th run average time : 1692
21th run average time : 1741
22th run average time : 1930
23th run average time : 1698
24th run average time : 1488
25th run average time : 1707
26th run average time : 1807
27th run average time : 2092
stopped the test application with closing the connection and started with a
new ES connection:
1th run average time: 417ms
2th run average time: 1487ms
3th run average time: 1588
4th run average time: 1573
5th run average time: 1681
6th run average time: 1721
7th run average time: 1962
8th run average time: 1751
9th run average time: 1607
10th run average time: 1801
11th run average time: 1660
And this behavior occurs every time I restart the test, the first
executions are faster than before. So the only thing I could imagine was
the long time open connection.
Thanks ,
Vahid Hassani
On Friday, August 30, 2013 9:46:18 AM UTC+2, Adrien Grand wrote:
Hi,
Are you sure that this is due to long-living connections? I'm asking
because there are lots of moving parts. For example, it could be that
search became slower because large background merges were running and they
had finished before you started the test again? Otherwise, I have no clue
why a long-living connection would have worse performance than a newly-open
one.
On Thu, Aug 29, 2013 at 6:05 PM, Vahid <vhas...@gmail.com <javascript:>>wrote:
Hi ,
I've created a stress test to submit 100 concurrent queries to ES
continuesly. After sending 100 requests, waits 1-2 minutes and send the
next 100 requests.
ES configured on a single node with one shard and no replica.
There is only one single ES connection to submit all the queries for all
the runs. Also documents are indexing constantly (1300 docs/s).
After near half an hour, the average search speed decreases to half. If I
stop the test and start it (connection will be closed and a new one
openned) speed will be fine again.
There is no performance problem with indexing.
So my question is, what could be the reason of search speed decrease.
index configuration:
index.compound_format: false
index.version.created: 900399
index.number_of_shards: 1
index.translog.flush_threshold_period: 60s
index.store.throttle.type: merge
index.cache.filter.max_size: 10
index.cache.field.expire: 1m
index.merge.policy.merge_factor: 30
index.cache.filter.expire: 1m
index.number_of_replicas: 1
index.store.throttle.max_bytes_per_sec: 5mb
index.merge.policy.use_compound_file: false
index.store.compress.stored: true
index.cache.field.type: resident
index.indices.memory.index_buffer_size: 20%
ES version is 0.90.3 and I'm using TransportClient to get connections*.*
Thanks and Best regards,
Vahid
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
You did not tell us about the nature of your documents and queries and your
heap size. Also the JVM version is significant.
In other words, the behaviour is not surprising on a single node. If you
connect with 100 threads to a node, this node will allocate very much
resources to answer your queries quickly. After 1-2 minutes, and indexing
with 1300 dps, theres resources are gone, they are now allocated by the
indexer. To answer 100 concurrent queries again, the node must shovel
harder for resources. The longer the indexing takes, the more pressure is
there on the heap, and the longer it takes to answer queries.
Take more nodes and you will see how ES distributes the load better.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.