# Search speed decrease with long time open connection **URL:** \ **Category:** Elasticsearch\ **Created:** [August 29, 2013, 4:05pm UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395 "2013-08-29T16:05:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [August 29, 2013, 4:05pm UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395/1 "2013-08-29T16:05:23Z")
Hi , I've created a stress test to submit 100 concurrent queries to ES continuesly. After sending 100 requests, waits 1-2 minutes and send the next 100 requests. ES configured on a single node with one shard and no replica. There is only one single ES connection to submit all the queries for all the runs. Also documents are indexing constantly (1300 docs/s). After near half an hour, the average search speed decreases to half. If I stop the test and start it (connection will be closed and a new one openned) speed will be fine again. There is no performance problem with indexing. So my question is, what could be the reason of search speed decrease. index configuration: - index.compound\_format: false - index.version.created: 900399 - index.number\_of\_shards: 1 - index.translog.flush\_threshold\_period: 60s - index.store.throttle.type: merge - index.cache.filter.max\_size: 10 - index.cache.field.expire: 1m - index.merge.policy.merge\_factor: 30 - index.cache.filter.expire: 1m - index.number\_of\_replicas: 1 - index.store.throttle.max\_bytes\_per\_sec: 5mb - index.merge.policy.use\_compound\_file: false - index.store.compress.stored: true - index.cache.field.type: resident - index.indices.memory.index\_buffer\_size: 20% ES version is 0.90.3 and I'm using TransportClient to get connections\*.\* Thanks and Best regards, Vahid -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 30, 2013, 7:46am UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395/2 "2013-08-30T07:46:18Z")
Hi, Are you sure that this is due to long-living connections? I'm asking because there are lots of moving parts. For example, it could be that search became slower because large background merges were running and they had finished before you started the test again? Otherwise, I have no clue why a long-living connection would have worse performance than a newly-open one. On Thu, Aug 29, 2013 at 6:05 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Hi , > > I've created a stress test to submit 100 concurrent queries to ES > continuesly. After sending 100 requests, waits 1-2 minutes and send the > next 100 requests. > ES configured on a single node with one shard and no replica. > > There is only one single ES connection to submit all the queries for all > the runs. Also documents are indexing constantly (1300 docs/s). > After near half an hour, the average search speed decreases to half. If I > stop the test and start it (connection will be closed and a new one > openned) speed will be fine again. > There is no performance problem with indexing. > > So my question is, what could be the reason of search speed decrease. > > index configuration: > > - index.compound\_format: false > - index.version.created: 900399 > - index.number\_of\_shards: 1 > - index.translog.flush\_threshold\_period: 60s > - index.store.throttle.type: merge > - index.cache.filter.max\_size: 10 > - index.cache.field.expire: 1m > - index.merge.policy.merge\_factor: 30 > - index.cache.filter.expire: 1m > - index.number\_of\_replicas: 1 > - index.store.throttle.max\_bytes\_per\_sec: 5mb > - index.merge.policy.use\_compound\_file: false > - index.store.compress.stored: true > - index.cache.field.type: resident > - index.indices.memory.index\_buffer\_size: 20% > > ES version is 0.90.3 and I'm using TransportClient to get connections\*.\* > > Thanks and Best regards, > Vahid > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [August 30, 2013, 10:07am UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395/3 "2013-08-30T10:07:34Z")
Hi, thank you Adrien the average of response time for 100 queries at each run are written, the times are in ms: _1th run average time : 371 2th run average time : 358_ 3th run average time : 1479 4th run average time : 1321 5th run average time : 1661 6th run average time : 1741 7th run average time : 1768 8th run average time : 1928 9th run average time : 1638 10th run average time : 1396 11th run average time : 1682 12th run average time : 1355 13th run average time : 1599 14th run average time : 1250 15th run average time : 1387 16th run average time : 1364 17th run average time : 1044 18th run average time : 1313 19th run average time : 1726 20th run average time : 1692 21th run average time : 1741 22th run average time : 1930 23th run average time : 1698 24th run average time : 1488 25th run average time : 1707 26th run average time : 1807 27th run average time : 2092 stopped the test application with closing the connection and started with a new ES connection: _1th run average time: 417_ms 2th run average time: 1487ms 3th run average time: 1588 4th run average time: 1573 5th run average time: 1681 6th run average time: 1721 7th run average time: 1962 8th run average time: 1751 9th run average time: 1607 10th run average time: 1801 11th run average time: 1660 And this behavior occurs every time I restart the test, the first executions are faster than before. So the only thing I could imagine was the long time open connection. Thanks , Vahid Hassani On Friday, August 30, 2013 9:46:18 AM UTC+2, Adrien Grand wrote: > Hi, > > Are you sure that this is due to long-living connections? I'm asking > because there are lots of moving parts. For example, it could be that > search became slower because large background merges were running and they > had finished before you started the test again? Otherwise, I have no clue > why a long-living connection would have worse performance than a newly-open > one. > > On Thu, Aug 29, 2013 at 6:05 PM, Vahid \<[vhas...@gmail.com](mailto:vhas...@gmail.com) \\>wrote: > > > Hi , > > > > I've created a stress test to submit 100 concurrent queries to ES > > continuesly. After sending 100 requests, waits 1-2 minutes and send the > > next 100 requests. > > ES configured on a single node with one shard and no replica. > > > > There is only one single ES connection to submit all the queries for all > > the runs. Also documents are indexing constantly (1300 docs/s). > > After near half an hour, the average search speed decreases to half. If I > > stop the test and start it (connection will be closed and a new one > > openned) speed will be fine again. > > There is no performance problem with indexing. > > > > So my question is, what could be the reason of search speed decrease. > > > > index configuration: > > > > - index.compound\_format: false > > - index.version.created: 900399 > > - index.number\_of\_shards: 1 > > - index.translog.flush\_threshold\_period: 60s > > - index.store.throttle.type: merge > > - index.cache.filter.max\_size: 10 > > - index.cache.field.expire: 1m > > - index.merge.policy.merge\_factor: 30 > > - index.cache.filter.expire: 1m > > - index.number\_of\_replicas: 1 > > - index.store.throttle.max\_bytes\_per\_sec: 5mb > > - index.merge.policy.use\_compound\_file: false > > - index.store.compress.stored: true > > - index.cache.field.type: resident > > - index.indices.memory.index\_buffer\_size: 20% > > > > ES version is 0.90.3 and I'm using TransportClient to get connections\*.\* > > > > Thanks and Best regards, > > Vahid > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Adrien Grand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 30, 2013, 11:17pm UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395/4 "2013-08-30T23:17:28Z")
You did not tell us about the nature of your documents and queries and your heap size. Also the JVM version is significant. In other words, the behaviour is not surprising on a single node. If you connect with 100 threads to a node, this node will allocate very much resources to answer your queries quickly. After 1-2 minutes, and indexing with 1300 dps, theres resources are gone, they are now allocated by the indexer. To answer 100 concurrent queries again, the node must shovel harder for resources. The longer the indexing takes, the more pressure is there on the heap, and the longer it takes to answer queries. Take more nodes and you will see how ES distributes the load better. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:18am UTC](https://discuss.elastic.co/t/search-speed-decrease-with-long-time-open-connection/13395/5 "2017-07-06T02:18:41Z")