# Our Elastic search server can not serve many connections for a long time **URL:** \ **Category:** Elasticsearch\ **Created:** [March 30, 2015, 7:58am UTC](https://discuss.elastic.co/t/our-elastic-search-server-can-not-serve-many-connections-for-a-long-time/22975 "2015-03-30T07:58:58Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![quydo\_x](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quydo_x/32/792_2.png) [@quydo\_x](https://discuss.elastic.co/u/quydo_x)\ **Post date:** [March 30, 2015, 7:58am UTC](https://discuss.elastic.co/t/our-elastic-search-server-can-not-serve-many-connections-for-a-long-time/22975/1 "2015-03-30T07:58:58Z")
Hi all I am running Elastic Search in a dedicated server: 4 vCPU, 8GB RAM in Amazon Cloud. I installed ES version 1.5. In web application, I have cached in search query, although my ES server will be respond for many connections at the same time, I think about 20 I write a PHP library interact with ES server via curl such as curl -XGET http://elastic\_server\_ip:9200/... Here is query in web application: $query = array( 'function\_score' =\> array( 'query' =\> array( 'bool' =\> array( 'must' =\> array( array( 'multi\_match' =\> array( 'query' =\> $keyword, 'fields' =\> array('name', 'vendor'), 'operator' =\> 'or' ) ), array('term' =\> array('publish' =\> 1)), array('term' =\> array('app\_type' =\> $this-\> store)) ) ) ), 'script\_score' =\> array( 'script' =\> "\_score + doc['rank'].value" ) ) ``` ); */ $params = array( 'fields' => array('_id'), "query" => $query, "size" => $this->limit, "from" => $this->start, "sort" => array( "_score" => "desc" ) ); ``` _In /etc/init.d/elasticsearch_ (init script) es\_heap\_size: 2G es\_heap\_newsize: es\_min\_mem: 512M es\_max\_mem: 1G max\_locked\_memory: unlimited _In /etc/elasticsearch/elasticsearch.yml_, all settings are default except these following settings: script.groovy.sandbox.enabled: true bootstrap.mlockall: true threadpool.index.type: fixed threadpool.index.size: 4 threadpool.index.queue\_size: 1000 threadpool.search.queue\_size: 1000 threadpool.search.type: cached threadpool.bulk.type: fixed threadpool.bulk.size: 4 # availableProcessors threadpool.bulk.queue\_size: 1000 With these settings, The ES server will run normally for a long time (about 1 hours) But after that time, the ES server don't response any more connections. If I reduce the HEAP SIZE to 1G, I think the time that ES server run normally is about 10 minutes Here is some log when ES start: [2015-03-30 06:59:05,877][DEBUG][gateway.local.state.shards] [Arkus] [ my\_index][3] shard state info found: [version [10], primary [true]] [2015-03-30 06:59:05,877][DEBUG][gateway.local] [Arkus] [ my\_index][3]: throttling allocation [[my\_index][3], node[null], [P], s[ UNASSIGNED]] to [[[Arkus][RQCnuFvBQbypN5GCUa04zw][ip-10-0-0-230][inet[/10.0.0.230:9300]]]] on primary allocation [2015-03-30 06:59:05,880][DEBUG][cluster.service] [Arkus] cluster state updated, version [2], source [local-gateway-elected-state] [2015-03-30 06:59:05,881][DEBUG][cluster.service] [Arkus] publishing cluster state version 2 [2015-03-30 06:59:05,881][DEBUG][cluster.service] [Arkus] set local cluster state to version 2 [2015-03-30 06:59:05,881][DEBUG][indices.cluster] [Arkus] [my\_index] creating index [2015-03-30 06:59:05,882][DEBUG][indices] [Arkus] creating Index [my\_index], shards [5]/[1] [2015-03-30 06:59:06,078][DEBUG][index.mapper] [Arkus] [ my\_index] using dynamic[true], default mapping: default\_mapping\_location[ null], loaded\_from[jar:file:/usr/share/elasticsearch/lib/elasticsearch-1.5. 0.jar!/org/elasticsearch/index/mapper/default-mapping.json], default percolator mapping: location[null], loaded\_from[null] [2015-03-30 06:59:06,078][DEBUG][index.cache.query.parser.resident] [Arkus] [my\_index] using [resident] query cache with max\_size [100], expire [null] [2015-03-30 06:59:06,082][DEBUG][index.store.fs] [Arkus] [ my\_index] using index.store.throttle.type [node], with index.store.throttle.max\_bytes\_per\_sec [0b] [2015-03-30 06:59:06,126][DEBUG][action.search.type] [Arkus] All shards failed for phase: [query] org.elasticsearch.indices.IndexMissingException: [my\_index] missing at org.elasticsearch.indices.IndicesService.indexServiceSafe( IndicesService.java:284) at org.elasticsearch.search.SearchService.createContext( SearchService.java:544) at org.elasticsearch.search.SearchService.createAndPutContext( SearchService.java:529) at org.elasticsearch.search.SearchService.executeQueryPhase( SearchService.java:291) at org.elasticsearch.search.action.SearchServiceTransportAction$5. call(SearchServiceTransportAction.java:231) at org.elasticsearch.search.action.SearchServiceTransportAction$5. call(SearchServiceTransportAction.java:228) at org.elasticsearch.search.action.SearchServiceTransportAction$23. run(SearchServiceTransportAction.java:559) at java.util.concurrent.ThreadPoolExecutor.runWorker( ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run( ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2015-03-30 06:59:06,134][DEBUG][indices.cluster] [Arkus] [ my\_index] adding mapping [apps], source [{"apps":{"properties":{"app\_type":{ "type":"string","index\_analyzer":"index\_analyzer"},"name":{"type":"string", "index\_analyzer":"index\_analyzer","search\_analyzer":"search\_name\_analyzer", "search\_quote\_analyzer":"index\_analyzer"},"pubish":{"type":"integer"}, "publish":{"type":"long"},"rank":{"type":"double"},"vendor":{"type":"string" ,"index\_analyzer":"index\_analyzer","search\_analyzer":"search\_name\_analyzer", "search\_quote\_analyzer":"index\_analyzer"}}}}] [2015-03-30 06:59:06,171][DEBUG][action.search.type] [Arkus] All shards failed for phase: [query] org.elasticsearch.index.IndexShardMissingException: [my\_index][0] missing at org.elasticsearch.index.IndexService.shardSafe(IndexService.java: 210) at org.elasticsearch.search.SearchService.createContext( SearchService.java:545) at org.elasticsearch.search.SearchService.createAndPutContext( SearchService.java:529) _And here are the log between normal time and no response time_: [2015-03-30 06:59:07,845][DEBUG][script] [Arkus] notifying script services of script removal due to: [REPLACED] [2015-03-30 06:59:07,865][DEBUG][script] [Arkus] notifying script services of script removal due to: [REPLACED] [2015-03-30 06:59:15,849][DEBUG][cluster.service] [Arkus] processing [routing-table-updater]: execute [2015-03-30 06:59:15,850][DEBUG][cluster.service] [Arkus] processing [routing-table-updater]: no change in cluster\_state [2015-03-30 06:59:29,711][DEBUG][http.netty] [Arkus] Caught exception while handling client http traffic, closing connection [id: 0x6ca35b4e, /10.0.0.166:36271 :\> /10.0.0.230:9200] java.nio.channels.ClosedChannelException at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:199) at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:149) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2015-03-30 06:59:34,811][DEBUG][indices.memory] [Arkus] recalculating shard indexing buffer (reason=[[ADDED]]), total is [201.4mb] with [5] active shards, each shard set to indexing=[40.2mb], translog=[64kb [2015-03-30 07:05:12,474][DEBUG][http.netty] [Madame Menace] Caught exception while handling client http traffic, closing connection [id: 0x31d02618, /10.0.0.166:59967 :\> /10.0.0.230:9200] java.nio.channels.ClosedChannelException at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:128) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99) at org.elasticsearch.common.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779) at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:199) at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.doRun(TransportSearchQueryThenFetchAction.java:149) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [2015-03-30 07:05:18,046][INFO][monitor.jvm] [Madame Menace] [gc][old][97][32] duration [5.6s], collections [1]/[5.8s], total [5.6s]/[2m], memory [1.7gb]-\>[1.8gb]/[1.9gb], all\_pools {[young] [100.6mb]-\>[164.7mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [1.6gb]-\>[1.6gb]/[1.6gb]} [2015-03-30 07:05:21,899][DEBUG][monitor.jvm] [Madame Menace] [gc][old][98][33] duration [3.7s], collections [1]/[3.8s], total [3.7s]/[2m], memory [1.8gb]-\>[1.7gb]/[1.9gb], all\_pools {[young] [164.7mb]-\>[103mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [1.6gb]-\>[1.6gb]/[1.6gb]} [2015-03-30 07:05:27,746][INFO][monitor.jvm] [Madame Menace] [gc][old][99][34] duration [5.6s], collections [1]/[5.8s], total [5.6s]/[2.1m], memory [1.7gb]-\>[1.7gb]/[1.9gb], all\_pools {[young] [103mb]-\>[72.4mb]/[266.2mb]}{[survivor] [0b]-\>[0b]/[33.2mb]}{[old] [1.6gb]-\>[1.6gb]/[1.6gb]} Do you have any suggestion for that problem! Thank you very much -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e98ed453-ab08-408b-b957-390ed0bf8103%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e98ed453-ab08-408b-b957-390ed0bf8103%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:23am UTC](https://discuss.elastic.co/t/our-elastic-search-server-can-not-serve-many-connections-for-a-long-time/22975/2 "2017-07-06T00:23:09Z")