# Many slow query with high load after a hour

**URL:** <https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162>\
**Category:** Elasticsearch\
**Created:** [December 15, 2011, 5:55am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162 "2011-12-15T05:55:18Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 15, 2011, 5:55am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/1 "2011-12-15T05:55:18Z")

</div>

I test my program with high pressure of 1500 request per second and  
each request need to do a es query(i use MatchAllQuery and a bunch of  
filters as filters can be cached). The size of the index is 106.7kb  
with 200 documents. I start my es with paramters: bin/elasticsearch -  
Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true

After a hour, es begins to become slow for query and from log i can  
see lots of slow query, i paste some logs below and wish your help:

[2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
[Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
{"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
{"lcs":"recommendation.test.lc.3"}},"must":{"term":  
{"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
{"status":1}},"must":{"range":{"from":{"from":null,"to":  
1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
{"range":{"to":{"from":  
1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must":  
{"range":{"hMax":{"from":  
800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
{"range":{"hMin":{"from":null,"to":  
800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
{"wMax":{"from":  
480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
{"range":{"wMin":{"from":null,"to":  
480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields":"id"}  
extra\_source[],

the mapping are:  
{  
"test":{  
"\_all" : {  
"enabled" : false  
},  
"properties" : {  
"id":{  
"type":"string",  
"index":"not\_analyzed",  
"search\_analyzer":"keyword",  
"store":"yes"  
},  
"pkgs":{  
"type":"string",  
"index":"not\_analyzed",  
"search\_analyzer":"keyword",  
"store":"no"  
},  
"lcs":{  
"type":"string",  
"index":"not\_analyzed",  
"search\_analyzer":"keyword",  
"store":"no"  
},  
"from":{  
"type":"long",  
"index":"not\_analyzed",  
"store":"no"  
},  
"to":{  
"type":"long",  
"index":"not\_analyzed",  
"store":"no"  
},  
"status":{  
"type":"integer",  
"index":"not\_analyzed",  
"store":"no"  
},  
"nets" : {  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"androidAPILevels":{  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"hMin":{  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"hMax":{  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"wMin":{  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"wMax":{  
"type" : "integer",  
"index" : "not\_analyzed",  
"store":"no"  
},  
"models":{  
"type" : "string",  
"index": "analyzed",  
"index\_analyzer":"standardAnalyzer",  
"search\_analyzer":"standardAnalyzer",  
"store":"no"  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 15, 2011, 5:59am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/2 "2011-12-15T05:59:47Z")

</div>

another problem is though there is only 200 documents and 100k data  
storage, the memory use under pressure is more than 2g(with top under  
linux). I don't know why so much memory is needed.

On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:

> I test my program with high pressure of 1500 request per second and  
> each request need to do a es query(i use MatchAllQuery and a bunch of  
> filters as filters can be cached). The size of the index is 106.7kb  
> with 200 documents. I start my es with paramters: bin/elasticsearch -  
> Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> After a hour, es begins to become slow for query and from log i can  
> see lots of slow query, i paste some logs below and wish your help:
> 
> [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> {"range":{"to":{"from":  
> 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> {"range":{"hMax":{"from":  
> 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> {"range":{"hMin":{"from":null,"to":  
> 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> {"wMax":{"from":  
> 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> {"range":{"wMin":{"from":null,"to":  
> 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> extra\_source,
> 
> the mapping are:  
> {  
> "test":{  
> "\_all" : {  
> "enabled" : false  
> },  
> "properties" : {  
> "id":{  
> "type":"string",  
> "index":"not\_analyzed",  
> "search\_analyzer":"keyword",  
> "store":"yes"  
> },  
> "pkgs":{  
> "type":"string",  
> "index":"not\_analyzed",  
> "search\_analyzer":"keyword",  
> "store":"no"  
> },  
> "lcs":{  
> "type":"string",  
> "index":"not\_analyzed",  
> "search\_analyzer":"keyword",  
> "store":"no"  
> },  
> "from":{  
> "type":"long",  
> "index":"not\_analyzed",  
> "store":"no"  
> },  
> "to":{  
> "type":"long",  
> "index":"not\_analyzed",  
> "store":"no"  
> },  
> "status":{  
> "type":"integer",  
> "index":"not\_analyzed",  
> "store":"no"  
> },  
> "nets" : {  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "androidAPILevels":{  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "hMin":{  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "hMax":{  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "wMin":{  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "wMax":{  
> "type" : "integer",  
> "index" : "not\_analyzed",  
> "store":"no"  
> },  
> "models":{  
> "type" : "string",  
> "index": "analyzed",  
> "index\_analyzer":"standardAnalyzer",  
> "search\_analyzer":"standardAnalyzer",  
> "store":"no"  
> }  
> }  
> }
> 
> }

---

<div class="post-metadata">

**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [December 15, 2011, 8:23am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/3 "2011-12-15T08:23:53Z")

</div>

you can try to give it a lot less memory (e.g. \<50MB ?) and then  
garbage collector will be called more often.

not sure why ES will take so much time after an hour. how many shards  
do you have + how many CPUs? also try to monitor with ES-DESK or  
jvisualvm to see the real RAM usage and CPU load while testing.

Peter.

On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:

> another problem is though there is only 200 documents and 100k data  
> storage, the memory use under pressure is more than 2g(with top under  
> linux). I don't know why so much memory is needed.
> 
> On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > I test my program with high pressure of 1500 request per second and  
> > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > filters as filters can be cached). The size of the index is 106.7kb  
> > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> > After a hour, es begins to become slow for query and from log i can  
> > see lots of slow query, i paste some logs below and wish your help:
> 
> > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > {"range":{"to":{"from":  
> > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > {"range":{"hMax":{"from":  
> > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > {"range":{"hMin":{"from":null,"to":  
> > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > {"wMax":{"from":  
> > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > {"range":{"wMin":{"from":null,"to":  
> > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > extra\_source,
> 
> > the mapping are:  
> > {  
> > "test":{  
> > "\_all" : {  
> > "enabled" : false  
> > },  
> > "properties" : {  
> > "id":{  
> > "type":"string",  
> > "index":"not\_analyzed",  
> > "search\_analyzer":"keyword",  
> > "store":"yes"  
> > },  
> > "pkgs":{  
> > "type":"string",  
> > "index":"not\_analyzed",  
> > "search\_analyzer":"keyword",  
> > "store":"no"  
> > },  
> > "lcs":{  
> > "type":"string",  
> > "index":"not\_analyzed",  
> > "search\_analyzer":"keyword",  
> > "store":"no"  
> > },  
> > "from":{  
> > "type":"long",  
> > "index":"not\_analyzed",  
> > "store":"no"  
> > },  
> > "to":{  
> > "type":"long",  
> > "index":"not\_analyzed",  
> > "store":"no"  
> > },  
> > "status":{  
> > "type":"integer",  
> > "index":"not\_analyzed",  
> > "store":"no"  
> > },  
> > "nets" : {  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "androidAPILevels":{  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "hMin":{  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "hMax":{  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "wMin":{  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "wMax":{  
> > "type" : "integer",  
> > "index" : "not\_analyzed",  
> > "store":"no"  
> > },  
> > "models":{  
> > "type" : "string",  
> > "index": "analyzed",  
> > "index\_analyzer":"standardAnalyzer",  
> > "search\_analyzer":"standardAnalyzer",  
> > "store":"no"  
> > }  
> > }  
> > }
> 
> > }

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 15, 2011, 9:03am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/4 "2011-12-15T09:03:00Z")

</div>

only one elasticsearch instance with one shard, 0 replica. I tested  
another time with 4g maximum memeory, after one hour, the problem  
occurs again. I stop jmeter and use top to monitor the es process and  
found the memory usage stays on 4.2g and cpu load jump from 20+% to  
300%. I was considering using es in my program for a high load web  
service project, now i have to consider using pure lucene.

my es config is shown as:  
{  
"cluster":{  
"name":"es-cluster"  
},  
"gateway":{  
"recover\_after\_nodes": 1,  
"recover\_after\_time": "30s",  
"expected\_nodes": 2  
},  
"network":  
{  
"host":"0.0.0.0",  
"tcp":{  
"keep\_alive":true,  
"send\_buffer\_size":"50m",  
"receive\_buffer\_size":"50m"  
}  
},  
"transport":{  
"tcp":{  
"port":"9350-9400",  
"keep\_alive":true,  
"send\_buffer\_size":"20m",  
"receive\_buffer\_size":"20m",  
"connect\_timeout":"5s"  
}  
},  
"http":{  
"port":"9250-9300"  
},  
"index" : {  
"store":{  
"cache":{  
"memory":{  
"small\_buffer\_size":"32mb",  
"large\_buffer\_size":"64mb",  
"small\_cache\_size":"512mb",  
"large\_cache\_size":"1g"  
}  
}  
},  
"search":{  
"slowlog":{  
"threshold":{  
"query":{  
"warn":"1s",  
"info":"500ms",  
"debug":"200ms",  
"trace":"50ms"  
},  
"fetch":{  
"warn":"100ms",  
"info":"80ms",  
"debug":"50ms",  
"trace":"20ms"  
}  
}  
}  
},  
"number\_of\_shards":2,  
"number\_of\_replicas":1,  
"refresh\_interval":"1s",  
"term\_index\_interval":64,  
"analysis" : {  
"analyzer" : {  
"nGramAnalyzer":{  
"type":"custom",  
"tokenizer":"standard",  
"filter":  
["standard","lowercase","englishSnowball","nGramFilter"]  
},  
"standardAnalyzer":{  
"type":"custom",  
"tokenizer":"standard",  
"filter":  
["standard","lowercase","englishSnowball"]  
}  
},  
"filter":{  
"nGramFilter":{  
"type":"nGram",  
"min\_gram":1,  
"max\_gram":64  
},  
"edgeNGramFilter":{  
"type":"edgeNGram",  
"min\_gram":1,  
"max\_gram":64,  
"side":"front"  
},  
"englishSnowball":{  
"type":"snowball",  
"language":"English"  
}  
}  
}  
}  
}

On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:

> you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> garbage collector will be called more often.
> 
> not sure why ES will take so much time after an hour. how many shards  
> do you have + how many CPUs? also try to monitor with ES-DESK or  
> jvisualvm to see the real RAM usage and CPU load while testing.
> 
> Peter.
> 
> On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > another problem is though there is only 200 documents and 100k data  
> > storage, the memory use under pressure is more than 2g(with top under  
> > linux). I don't know why so much memory is needed.
> 
> > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > I test my program with high pressure of 1500 request per second and  
> > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > filters as filters can be cached). The size of the index is 106.7kb  
> > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> > > After a hour, es begins to become slow for query and from log i can  
> > > see lots of slow query, i paste some logs below and wish your help:
> 
> > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > {"range":{"to":{"from":  
> > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > {"range":{"hMax":{"from":  
> > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > {"range":{"hMin":{"from":null,"to":  
> > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > {"wMax":{"from":  
> > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > {"range":{"wMin":{"from":null,"to":  
> > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > extra\_source,
> 
> > > the mapping are:  
> > > {  
> > > "test":{  
> > > "\_all" : {  
> > > "enabled" : false  
> > > },  
> > > "properties" : {  
> > > "id":{  
> > > "type":"string",  
> > > "index":"not\_analyzed",  
> > > "search\_analyzer":"keyword",  
> > > "store":"yes"  
> > > },  
> > > "pkgs":{  
> > > "type":"string",  
> > > "index":"not\_analyzed",  
> > > "search\_analyzer":"keyword",  
> > > "store":"no"  
> > > },  
> > > "lcs":{  
> > > "type":"string",  
> > > "index":"not\_analyzed",  
> > > "search\_analyzer":"keyword",  
> > > "store":"no"  
> > > },  
> > > "from":{  
> > > "type":"long",  
> > > "index":"not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "to":{  
> > > "type":"long",  
> > > "index":"not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "status":{  
> > > "type":"integer",  
> > > "index":"not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "nets" : {  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "androidAPILevels":{  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "hMin":{  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "hMax":{  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "wMin":{  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "wMax":{  
> > > "type" : "integer",  
> > > "index" : "not\_analyzed",  
> > > "store":"no"  
> > > },  
> > > "models":{  
> > > "type" : "string",  
> > > "index": "analyzed",  
> > > "index\_analyzer":"standardAnalyzer",  
> > > "search\_analyzer":"standardAnalyzer",  
> > > "store":"no"  
> > > }  
> > > }  
> > > }
> 
> > > }

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [December 15, 2011, 12:00pm UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/5 "2011-12-15T12:00:17Z")

</div>

On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:

> only one elasticsearch instance with one shard, 0 replica. I tested  
> another time with 4g maximum memeory, after one hour, the problem  
> occurs again. I stop jmeter and use top to monitor the es process and  
> found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> 300%. I was considering using es in my program for a high load web  
> service project, now i have to consider using pure lucene.

Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
and ES\_MIN/MAX\_MEM correctly

There are numerous emails explaining how to do this in this list, so a  
quick search should find some

clint

> my es config is shown as:  
> {  
> "cluster":{  
> "name":"es-cluster"  
> },  
> "gateway":{  
> "recover\_after\_nodes": 1,  
> "recover\_after\_time": "30s",  
> "expected\_nodes": 2  
> },  
> "network":  
> {  
> "host":"0.0.0.0",  
> "tcp":{  
> "keep\_alive":true,  
> "send\_buffer\_size":"50m",  
> "receive\_buffer\_size":"50m"  
> }  
> },  
> "transport":{  
> "tcp":{  
> "port":"9350-9400",  
> "keep\_alive":true,  
> "send\_buffer\_size":"20m",  
> "receive\_buffer\_size":"20m",  
> "connect\_timeout":"5s"  
> }  
> },  
> "http":{  
> "port":"9250-9300"  
> },  
> "index" : {  
> "store":{  
> "cache":{  
> "memory":{  
> "small\_buffer\_size":"32mb",  
> "large\_buffer\_size":"64mb",  
> "small\_cache\_size":"512mb",  
> "large\_cache\_size":"1g"  
> }  
> }  
> },  
> "search":{  
> "slowlog":{  
> "threshold":{  
> "query":{  
> "warn":"1s",  
> "info":"500ms",  
> "debug":"200ms",  
> "trace":"50ms"  
> },  
> "fetch":{  
> "warn":"100ms",  
> "info":"80ms",  
> "debug":"50ms",  
> "trace":"20ms"  
> }  
> }  
> }  
> },  
> "number\_of\_shards":2,  
> "number\_of\_replicas":1,  
> "refresh\_interval":"1s",  
> "term\_index\_interval":64,  
> "analysis" : {  
> "analyzer" : {  
> "nGramAnalyzer":{  
> "type":"custom",  
> "tokenizer":"standard",  
> "filter":  
> ["standard","lowercase","englishSnowball","nGramFilter"]  
> },  
> "standardAnalyzer":{  
> "type":"custom",  
> "tokenizer":"standard",  
> "filter":  
> ["standard","lowercase","englishSnowball"]  
> }  
> },  
> "filter":{  
> "nGramFilter":{  
> "type":"nGram",  
> "min\_gram":1,  
> "max\_gram":64  
> },  
> "edgeNGramFilter":{  
> "type":"edgeNGram",  
> "min\_gram":1,  
> "max\_gram":64,  
> "side":"front"  
> },  
> "englishSnowball":{  
> "type":"snowball",  
> "language":"English"  
> }  
> }  
> }  
> }  
> }
> 
> On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> 
> > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > garbage collector will be called more often.
> > 
> > not sure why ES will take so much time after an hour. how many shards  
> > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > jvisualvm to see the real RAM usage and CPU load while testing.
> > 
> > Peter.
> > 
> > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > another problem is though there is only 200 documents and 100k data  
> > > storage, the memory use under pressure is more than 2g(with top under  
> > > linux). I don't know why so much memory is needed.
> > 
> > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > I test my program with high pressure of 1500 request per second and  
> > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> > 
> > > > After a hour, es begins to become slow for query and from log i can  
> > > > see lots of slow query, i paste some logs below and wish your help:
> > 
> > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > {"range":{"to":{"from":  
> > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > {"range":{"hMax":{"from":  
> > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > {"range":{"hMin":{"from":null,"to":  
> > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > {"wMax":{"from":  
> > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > {"range":{"wMin":{"from":null,"to":  
> > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > extra\_source,
> > 
> > > > the mapping are:  
> > > > {  
> > > > "test":{  
> > > > "\_all" : {  
> > > > "enabled" : false  
> > > > },  
> > > > "properties" : {  
> > > > "id":{  
> > > > "type":"string",  
> > > > "index":"not\_analyzed",  
> > > > "search\_analyzer":"keyword",  
> > > > "store":"yes"  
> > > > },  
> > > > "pkgs":{  
> > > > "type":"string",  
> > > > "index":"not\_analyzed",  
> > > > "search\_analyzer":"keyword",  
> > > > "store":"no"  
> > > > },  
> > > > "lcs":{  
> > > > "type":"string",  
> > > > "index":"not\_analyzed",  
> > > > "search\_analyzer":"keyword",  
> > > > "store":"no"  
> > > > },  
> > > > "from":{  
> > > > "type":"long",  
> > > > "index":"not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "to":{  
> > > > "type":"long",  
> > > > "index":"not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "status":{  
> > > > "type":"integer",  
> > > > "index":"not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "nets" : {  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "androidAPILevels":{  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "hMin":{  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "hMax":{  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "wMin":{  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "wMax":{  
> > > > "type" : "integer",  
> > > > "index" : "not\_analyzed",  
> > > > "store":"no"  
> > > > },  
> > > > "models":{  
> > > > "type" : "string",  
> > > > "index": "analyzed",  
> > > > "index\_analyzer":"standardAnalyzer",  
> > > > "search\_analyzer":"standardAnalyzer",  
> > > > "store":"no"  
> > > > }  
> > > > }  
> > > > }
> > 
> > > > }

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 15, 2011, 3:24pm UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/6 "2011-12-15T15:24:53Z")

</div>

i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
files=true -Dbootstrap.mlockall=true -p es.pid

On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> 
> > only one elasticsearch instance with one shard, 0 replica. I tested  
> > another time with 4g maximum memeory, after one hour, the problem  
> > occurs again. I stop jmeter and use top to monitor the es process and  
> > found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> > 300%. I was considering using es in my program for a high load web  
> > service project, now i have to consider using pure lucene.
> 
> Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
> and ES\_MIN/MAX\_MEM correctly
> 
> There are numerous emails explaining how to do this in this list, so a  
> quick search should find some
> 
> clint
> 
> > my es config is shown as:  
> > {  
> > "cluster":{  
> > "name":"es-cluster"  
> > },  
> > "gateway":{  
> > "recover\_after\_nodes": 1,  
> > "recover\_after\_time": "30s",  
> > "expected\_nodes": 2  
> > },  
> > "network":  
> > {  
> > "host":"0.0.0.0",  
> > "tcp":{  
> > "keep\_alive":true,  
> > "send\_buffer\_size":"50m",  
> > "receive\_buffer\_size":"50m"  
> > }  
> > },  
> > "transport":{  
> > "tcp":{  
> > "port":"9350-9400",  
> > "keep\_alive":true,  
> > "send\_buffer\_size":"20m",  
> > "receive\_buffer\_size":"20m",  
> > "connect\_timeout":"5s"  
> > }  
> > },  
> > "http":{  
> > "port":"9250-9300"  
> > },  
> > "index" : {  
> > "store":{  
> > "cache":{  
> > "memory":{  
> > "small\_buffer\_size":"32mb",  
> > "large\_buffer\_size":"64mb",  
> > "small\_cache\_size":"512mb",  
> > "large\_cache\_size":"1g"  
> > }  
> > }  
> > },  
> > "search":{  
> > "slowlog":{  
> > "threshold":{  
> > "query":{  
> > "warn":"1s",  
> > "info":"500ms",  
> > "debug":"200ms",  
> > "trace":"50ms"  
> > },  
> > "fetch":{  
> > "warn":"100ms",  
> > "info":"80ms",  
> > "debug":"50ms",  
> > "trace":"20ms"  
> > }  
> > }  
> > }  
> > },  
> > "number\_of\_shards":2,  
> > "number\_of\_replicas":1,  
> > "refresh\_interval":"1s",  
> > "term\_index\_interval":64,  
> > "analysis" : {  
> > "analyzer" : {  
> > "nGramAnalyzer":{  
> > "type":"custom",  
> > "tokenizer":"standard",  
> > "filter":  
> > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > },  
> > "standardAnalyzer":{  
> > "type":"custom",  
> > "tokenizer":"standard",  
> > "filter":  
> > ["standard","lowercase","englishSnowball"]  
> > }  
> > },  
> > "filter":{  
> > "nGramFilter":{  
> > "type":"nGram",  
> > "min\_gram":1,  
> > "max\_gram":64  
> > },  
> > "edgeNGramFilter":{  
> > "type":"edgeNGram",  
> > "min\_gram":1,  
> > "max\_gram":64,  
> > "side":"front"  
> > },  
> > "englishSnowball":{  
> > "type":"snowball",  
> > "language":"English"  
> > }  
> > }  
> > }  
> > }  
> > }
> 
> > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> > 
> > > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > > garbage collector will be called more often.
> 
> > > not sure why ES will take so much time after an hour. how many shards  
> > > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > > jvisualvm to see the real RAM usage and CPU load while testing.
> 
> > > Peter.
> 
> > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > another problem is though there is only 200 documents and 100k data  
> > > > storage, the memory use under pressure is more than 2g(with top under  
> > > > linux). I don't know why so much memory is needed.
> 
> > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > I test my program with high pressure of 1500 request per second and  
> > > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> > > > > After a hour, es begins to become slow for query and from log i can  
> > > > > see lots of slow query, i paste some logs below and wish your help:
> 
> > > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > > {"range":{"to":{"from":  
> > > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > > {"range":{"hMax":{"from":  
> > > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > {"range":{"hMin":{"from":null,"to":  
> > > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > > {"wMax":{"from":  
> > > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > {"range":{"wMin":{"from":null,"to":  
> > > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > > extra\_source,
> 
> > > > > the mapping are:  
> > > > > {  
> > > > > "test":{  
> > > > > "\_all" : {  
> > > > > "enabled" : false  
> > > > > },  
> > > > > "properties" : {  
> > > > > "id":{  
> > > > > "type":"string",  
> > > > > "index":"not\_analyzed",  
> > > > > "search\_analyzer":"keyword",  
> > > > > "store":"yes"  
> > > > > },  
> > > > > "pkgs":{  
> > > > > "type":"string",  
> > > > > "index":"not\_analyzed",  
> > > > > "search\_analyzer":"keyword",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "lcs":{  
> > > > > "type":"string",  
> > > > > "index":"not\_analyzed",  
> > > > > "search\_analyzer":"keyword",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "from":{  
> > > > > "type":"long",  
> > > > > "index":"not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "to":{  
> > > > > "type":"long",  
> > > > > "index":"not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "status":{  
> > > > > "type":"integer",  
> > > > > "index":"not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "nets" : {  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "androidAPILevels":{  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "hMin":{  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "hMax":{  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "wMin":{  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "wMax":{  
> > > > > "type" : "integer",  
> > > > > "index" : "not\_analyzed",  
> > > > > "store":"no"  
> > > > > },  
> > > > > "models":{  
> > > > > "type" : "string",  
> > > > > "index": "analyzed",  
> > > > > "index\_analyzer":"standardAnalyzer",  
> > > > > "search\_analyzer":"standardAnalyzer",  
> > > > > "store":"no"  
> > > > > }  
> > > > > }  
> > > > > }
> 
> > > > > }

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [December 15, 2011, 3:42pm UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/7 "2011-12-15T15:42:46Z")

</div>

On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:

> i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> files=true -Dbootstrap.mlockall=true -p es.pid

you don't mention whether you are setting: ulimit -l unlimited

without that, mlockall won't work.

clint

> On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > 
> > > only one elasticsearch instance with one shard, 0 replica. I tested  
> > > another time with 4g maximum memeory, after one hour, the problem  
> > > occurs again. I stop jmeter and use top to monitor the es process and  
> > > found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> > > 300%. I was considering using es in my program for a high load web  
> > > service project, now i have to consider using pure lucene.
> > 
> > Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
> > and ES\_MIN/MAX\_MEM correctly
> > 
> > There are numerous emails explaining how to do this in this list, so a  
> > quick search should find some
> > 
> > clint
> > 
> > > my es config is shown as:  
> > > {  
> > > "cluster":{  
> > > "name":"es-cluster"  
> > > },  
> > > "gateway":{  
> > > "recover\_after\_nodes": 1,  
> > > "recover\_after\_time": "30s",  
> > > "expected\_nodes": 2  
> > > },  
> > > "network":  
> > > {  
> > > "host":"0.0.0.0",  
> > > "tcp":{  
> > > "keep\_alive":true,  
> > > "send\_buffer\_size":"50m",  
> > > "receive\_buffer\_size":"50m"  
> > > }  
> > > },  
> > > "transport":{  
> > > "tcp":{  
> > > "port":"9350-9400",  
> > > "keep\_alive":true,  
> > > "send\_buffer\_size":"20m",  
> > > "receive\_buffer\_size":"20m",  
> > > "connect\_timeout":"5s"  
> > > }  
> > > },  
> > > "http":{  
> > > "port":"9250-9300"  
> > > },  
> > > "index" : {  
> > > "store":{  
> > > "cache":{  
> > > "memory":{  
> > > "small\_buffer\_size":"32mb",  
> > > "large\_buffer\_size":"64mb",  
> > > "small\_cache\_size":"512mb",  
> > > "large\_cache\_size":"1g"  
> > > }  
> > > }  
> > > },  
> > > "search":{  
> > > "slowlog":{  
> > > "threshold":{  
> > > "query":{  
> > > "warn":"1s",  
> > > "info":"500ms",  
> > > "debug":"200ms",  
> > > "trace":"50ms"  
> > > },  
> > > "fetch":{  
> > > "warn":"100ms",  
> > > "info":"80ms",  
> > > "debug":"50ms",  
> > > "trace":"20ms"  
> > > }  
> > > }  
> > > }  
> > > },  
> > > "number\_of\_shards":2,  
> > > "number\_of\_replicas":1,  
> > > "refresh\_interval":"1s",  
> > > "term\_index\_interval":64,  
> > > "analysis" : {  
> > > "analyzer" : {  
> > > "nGramAnalyzer":{  
> > > "type":"custom",  
> > > "tokenizer":"standard",  
> > > "filter":  
> > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > },  
> > > "standardAnalyzer":{  
> > > "type":"custom",  
> > > "tokenizer":"standard",  
> > > "filter":  
> > > ["standard","lowercase","englishSnowball"]  
> > > }  
> > > },  
> > > "filter":{  
> > > "nGramFilter":{  
> > > "type":"nGram",  
> > > "min\_gram":1,  
> > > "max\_gram":64  
> > > },  
> > > "edgeNGramFilter":{  
> > > "type":"edgeNGram",  
> > > "min\_gram":1,  
> > > "max\_gram":64,  
> > > "side":"front"  
> > > },  
> > > "englishSnowball":{  
> > > "type":"snowball",  
> > > "language":"English"  
> > > }  
> > > }  
> > > }  
> > > }  
> > > }
> > 
> > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> > > 
> > > > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > > > garbage collector will be called more often.
> > 
> > > > not sure why ES will take so much time after an hour. how many shards  
> > > > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > > > jvisualvm to see the real RAM usage and CPU load while testing.
> > 
> > > > Peter.
> > 
> > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > > another problem is though there is only 200 documents and 100k data  
> > > > > storage, the memory use under pressure is more than 2g(with top under  
> > > > > linux). I don't know why so much memory is needed.
> > 
> > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > > > I test my program with high pressure of 1500 request per second and  
> > > > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> > 
> > > > > > After a hour, es begins to become slow for query and from log i can  
> > > > > > see lots of slow query, i paste some logs below and wish your help:
> > 
> > > > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > > > {"range":{"to":{"from":  
> > > > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > > > {"range":{"hMax":{"from":  
> > > > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > {"range":{"hMin":{"from":null,"to":  
> > > > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > > > {"wMax":{"from":  
> > > > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > {"range":{"wMin":{"from":null,"to":  
> > > > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > > > extra\_source,
> > 
> > > > > > the mapping are:  
> > > > > > {  
> > > > > > "test":{  
> > > > > > "\_all" : {  
> > > > > > "enabled" : false  
> > > > > > },  
> > > > > > "properties" : {  
> > > > > > "id":{  
> > > > > > "type":"string",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "search\_analyzer":"keyword",  
> > > > > > "store":"yes"  
> > > > > > },  
> > > > > > "pkgs":{  
> > > > > > "type":"string",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "search\_analyzer":"keyword",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "lcs":{  
> > > > > > "type":"string",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "search\_analyzer":"keyword",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "from":{  
> > > > > > "type":"long",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "to":{  
> > > > > > "type":"long",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "status":{  
> > > > > > "type":"integer",  
> > > > > > "index":"not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "nets" : {  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "androidAPILevels":{  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "hMin":{  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "hMax":{  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "wMin":{  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "wMax":{  
> > > > > > "type" : "integer",  
> > > > > > "index" : "not\_analyzed",  
> > > > > > "store":"no"  
> > > > > > },  
> > > > > > "models":{  
> > > > > > "type" : "string",  
> > > > > > "index": "analyzed",  
> > > > > > "index\_analyzer":"standardAnalyzer",  
> > > > > > "search\_analyzer":"standardAnalyzer",  
> > > > > > "store":"no"  
> > > > > > }  
> > > > > > }  
> > > > > > }
> > 
> > > > > > }

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 16, 2011, 2:29am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/8 "2011-12-16T02:29:14Z")

</div>

[2011-12-15 17:05:54,090][INFO][bootstrap]  
max\_open\_files[65510]  
i think it's enough.

I've already use es for another project with more than 2620000  
documents(10g+) but with low pressure. However, when i reindex all the  
documents i find es eat so much memory (5g+), so i set -Xms8g -Xmx8g  
for that project. I doubt that es has memory leak.

On Dec 15, 11:42 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:
> 
> > i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> > files=true -Dbootstrap.mlockall=true -p es.pid
> 
> you don't mention whether you are setting: ulimit -l unlimited
> 
> without that, mlockall won't work.
> 
> clint
> 
> > On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > > 
> > > > only one elasticsearch instance with one shard, 0 replica. I tested  
> > > > another time with 4g maximum memeory, after one hour, the problem  
> > > > occurs again. I stop jmeter and use top to monitor the es process and  
> > > > found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> > > > 300%. I was considering using es in my program for a high load web  
> > > > service project, now i have to consider using pure lucene.
> 
> > > Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
> > > and ES\_MIN/MAX\_MEM correctly
> 
> > > There are numerous emails explaining how to do this in this list, so a  
> > > quick search should find some
> 
> > > clint
> 
> > > > my es config is shown as:  
> > > > {  
> > > > "cluster":{  
> > > > "name":"es-cluster"  
> > > > },  
> > > > "gateway":{  
> > > > "recover\_after\_nodes": 1,  
> > > > "recover\_after\_time": "30s",  
> > > > "expected\_nodes": 2  
> > > > },  
> > > > "network":  
> > > > {  
> > > > "host":"0.0.0.0",  
> > > > "tcp":{  
> > > > "keep\_alive":true,  
> > > > "send\_buffer\_size":"50m",  
> > > > "receive\_buffer\_size":"50m"  
> > > > }  
> > > > },  
> > > > "transport":{  
> > > > "tcp":{  
> > > > "port":"9350-9400",  
> > > > "keep\_alive":true,  
> > > > "send\_buffer\_size":"20m",  
> > > > "receive\_buffer\_size":"20m",  
> > > > "connect\_timeout":"5s"  
> > > > }  
> > > > },  
> > > > "http":{  
> > > > "port":"9250-9300"  
> > > > },  
> > > > "index" : {  
> > > > "store":{  
> > > > "cache":{  
> > > > "memory":{  
> > > > "small\_buffer\_size":"32mb",  
> > > > "large\_buffer\_size":"64mb",  
> > > > "small\_cache\_size":"512mb",  
> > > > "large\_cache\_size":"1g"  
> > > > }  
> > > > }  
> > > > },  
> > > > "search":{  
> > > > "slowlog":{  
> > > > "threshold":{  
> > > > "query":{  
> > > > "warn":"1s",  
> > > > "info":"500ms",  
> > > > "debug":"200ms",  
> > > > "trace":"50ms"  
> > > > },  
> > > > "fetch":{  
> > > > "warn":"100ms",  
> > > > "info":"80ms",  
> > > > "debug":"50ms",  
> > > > "trace":"20ms"  
> > > > }  
> > > > }  
> > > > }  
> > > > },  
> > > > "number\_of\_shards":2,  
> > > > "number\_of\_replicas":1,  
> > > > "refresh\_interval":"1s",  
> > > > "term\_index\_interval":64,  
> > > > "analysis" : {  
> > > > "analyzer" : {  
> > > > "nGramAnalyzer":{  
> > > > "type":"custom",  
> > > > "tokenizer":"standard",  
> > > > "filter":  
> > > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > > },  
> > > > "standardAnalyzer":{  
> > > > "type":"custom",  
> > > > "tokenizer":"standard",  
> > > > "filter":  
> > > > ["standard","lowercase","englishSnowball"]  
> > > > }  
> > > > },  
> > > > "filter":{  
> > > > "nGramFilter":{  
> > > > "type":"nGram",  
> > > > "min\_gram":1,  
> > > > "max\_gram":64  
> > > > },  
> > > > "edgeNGramFilter":{  
> > > > "type":"edgeNGram",  
> > > > "min\_gram":1,  
> > > > "max\_gram":64,  
> > > > "side":"front"  
> > > > },  
> > > > "englishSnowball":{  
> > > > "type":"snowball",  
> > > > "language":"English"  
> > > > }  
> > > > }  
> > > > }  
> > > > }  
> > > > }
> 
> > > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> > > > 
> > > > > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > > > > garbage collector will be called more often.
> 
> > > > > not sure why ES will take so much time after an hour. how many shards  
> > > > > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > > > > jvisualvm to see the real RAM usage and CPU load while testing.
> 
> > > > > Peter.
> 
> > > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > > another problem is though there is only 200 documents and 100k data  
> > > > > > storage, the memory use under pressure is more than 2g(with top under  
> > > > > > linux). I don't know why so much memory is needed.
> 
> > > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > > > I test my program with high pressure of 1500 request per second and  
> > > > > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> > > > > > > After a hour, es begins to become slow for query and from log i can  
> > > > > > > see lots of slow query, i paste some logs below and wish your help:
> 
> > > > > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > > > > {"range":{"to":{"from":  
> > > > > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > > > > {"range":{"hMax":{"from":  
> > > > > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > {"range":{"hMin":{"from":null,"to":  
> > > > > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > > > > {"wMax":{"from":  
> > > > > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > {"range":{"wMin":{"from":null,"to":  
> > > > > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > > > > extra\_source,
> 
> > > > > > > the mapping are:  
> > > > > > > {  
> > > > > > > "test":{  
> > > > > > > "\_all" : {  
> > > > > > > "enabled" : false  
> > > > > > > },  
> > > > > > > "properties" : {  
> > > > > > > "id":{  
> > > > > > > "type":"string",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "search\_analyzer":"keyword",  
> > > > > > > "store":"yes"  
> > > > > > > },  
> > > > > > > "pkgs":{  
> > > > > > > "type":"string",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "search\_analyzer":"keyword",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "lcs":{  
> > > > > > > "type":"string",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "search\_analyzer":"keyword",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "from":{  
> > > > > > > "type":"long",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "to":{  
> > > > > > > "type":"long",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "status":{  
> > > > > > > "type":"integer",  
> > > > > > > "index":"not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "nets" : {  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "androidAPILevels":{  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "hMin":{  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "hMax":{  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "wMin":{  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "wMax":{  
> > > > > > > "type" : "integer",  
> > > > > > > "index" : "not\_analyzed",  
> > > > > > > "store":"no"  
> > > > > > > },  
> > > > > > > "models":{  
> > > > > > > "type" : "string",  
> > > > > > > "index": "analyzed",  
> > > > > > > "index\_analyzer":"standardAnalyzer",  
> > > > > > > "search\_analyzer":"standardAnalyzer",  
> > > > > > > "store":"no"  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }
> 
> > > > > > > }

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [December 16, 2011, 9:43am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/9 "2011-12-16T09:43:33Z")

</div>

On Thu, 2011-12-15 at 18:29 -0800, Weiwei Wang wrote:

> [2011-12-15 17:05:54,090][INFO][bootstrap]  
> max\_open\_files[65510]

ulimit -l is to do with locking memory, not open files. that is ulimit  
-n

clint

> i think it's enough.
> 
> I've already use es for another project with more than 2620000  
> documents(10g+) but with low pressure. However, when i reindex all the  
> documents i find es eat so much memory (5g+), so i set -Xms8g -Xmx8g  
> for that project. I doubt that es has memory leak.
> 
> On Dec 15, 11:42 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:
> > 
> > > i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> > > files=true -Dbootstrap.mlockall=true -p es.pid
> > 
> > you don't mention whether you are setting: ulimit -l unlimited
> > 
> > without that, mlockall won't work.
> > 
> > clint
> > 
> > > On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > 
> > > > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > > > 
> > > > > only one elasticsearch instance with one shard, 0 replica. I tested  
> > > > > another time with 4g maximum memeory, after one hour, the problem  
> > > > > occurs again. I stop jmeter and use top to monitor the es process and  
> > > > > found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> > > > > 300%. I was considering using es in my program for a high load web  
> > > > > service project, now i have to consider using pure lucene.
> > 
> > > > Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
> > > > and ES\_MIN/MAX\_MEM correctly
> > 
> > > > There are numerous emails explaining how to do this in this list, so a  
> > > > quick search should find some
> > 
> > > > clint
> > 
> > > > > my es config is shown as:  
> > > > > {  
> > > > > "cluster":{  
> > > > > "name":"es-cluster"  
> > > > > },  
> > > > > "gateway":{  
> > > > > "recover\_after\_nodes": 1,  
> > > > > "recover\_after\_time": "30s",  
> > > > > "expected\_nodes": 2  
> > > > > },  
> > > > > "network":  
> > > > > {  
> > > > > "host":"0.0.0.0",  
> > > > > "tcp":{  
> > > > > "keep\_alive":true,  
> > > > > "send\_buffer\_size":"50m",  
> > > > > "receive\_buffer\_size":"50m"  
> > > > > }  
> > > > > },  
> > > > > "transport":{  
> > > > > "tcp":{  
> > > > > "port":"9350-9400",  
> > > > > "keep\_alive":true,  
> > > > > "send\_buffer\_size":"20m",  
> > > > > "receive\_buffer\_size":"20m",  
> > > > > "connect\_timeout":"5s"  
> > > > > }  
> > > > > },  
> > > > > "http":{  
> > > > > "port":"9250-9300"  
> > > > > },  
> > > > > "index" : {  
> > > > > "store":{  
> > > > > "cache":{  
> > > > > "memory":{  
> > > > > "small\_buffer\_size":"32mb",  
> > > > > "large\_buffer\_size":"64mb",  
> > > > > "small\_cache\_size":"512mb",  
> > > > > "large\_cache\_size":"1g"  
> > > > > }  
> > > > > }  
> > > > > },  
> > > > > "search":{  
> > > > > "slowlog":{  
> > > > > "threshold":{  
> > > > > "query":{  
> > > > > "warn":"1s",  
> > > > > "info":"500ms",  
> > > > > "debug":"200ms",  
> > > > > "trace":"50ms"  
> > > > > },  
> > > > > "fetch":{  
> > > > > "warn":"100ms",  
> > > > > "info":"80ms",  
> > > > > "debug":"50ms",  
> > > > > "trace":"20ms"  
> > > > > }  
> > > > > }  
> > > > > }  
> > > > > },  
> > > > > "number\_of\_shards":2,  
> > > > > "number\_of\_replicas":1,  
> > > > > "refresh\_interval":"1s",  
> > > > > "term\_index\_interval":64,  
> > > > > "analysis" : {  
> > > > > "analyzer" : {  
> > > > > "nGramAnalyzer":{  
> > > > > "type":"custom",  
> > > > > "tokenizer":"standard",  
> > > > > "filter":  
> > > > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > > > },  
> > > > > "standardAnalyzer":{  
> > > > > "type":"custom",  
> > > > > "tokenizer":"standard",  
> > > > > "filter":  
> > > > > ["standard","lowercase","englishSnowball"]  
> > > > > }  
> > > > > },  
> > > > > "filter":{  
> > > > > "nGramFilter":{  
> > > > > "type":"nGram",  
> > > > > "min\_gram":1,  
> > > > > "max\_gram":64  
> > > > > },  
> > > > > "edgeNGramFilter":{  
> > > > > "type":"edgeNGram",  
> > > > > "min\_gram":1,  
> > > > > "max\_gram":64,  
> > > > > "side":"front"  
> > > > > },  
> > > > > "englishSnowball":{  
> > > > > "type":"snowball",  
> > > > > "language":"English"  
> > > > > }  
> > > > > }  
> > > > > }  
> > > > > }  
> > > > > }
> > 
> > > > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> > > > > 
> > > > > > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > > > > > garbage collector will be called more often.
> > 
> > > > > > not sure why ES will take so much time after an hour. how many shards  
> > > > > > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > > > > > jvisualvm to see the real RAM usage and CPU load while testing.
> > 
> > > > > > Peter.
> > 
> > > > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > > > > another problem is though there is only 200 documents and 100k data  
> > > > > > > storage, the memory use under pressure is more than 2g(with top under  
> > > > > > > linux). I don't know why so much memory is needed.
> > 
> > > > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > > > > > I test my program with high pressure of 1500 request per second and  
> > > > > > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > > > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > > > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > > > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> > 
> > > > > > > > After a hour, es begins to become slow for query and from log i can  
> > > > > > > > see lots of slow query, i paste some logs below and wish your help:
> > 
> > > > > > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > > > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > > > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > > > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > > > > > {"range":{"to":{"from":  
> > > > > > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > > > > > {"range":{"hMax":{"from":  
> > > > > > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > > {"range":{"hMin":{"from":null,"to":  
> > > > > > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > > > > > {"wMax":{"from":  
> > > > > > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > > {"range":{"wMin":{"from":null,"to":  
> > > > > > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > > > > > extra\_source,
> > 
> > > > > > > > the mapping are:  
> > > > > > > > {  
> > > > > > > > "test":{  
> > > > > > > > "\_all" : {  
> > > > > > > > "enabled" : false  
> > > > > > > > },  
> > > > > > > > "properties" : {  
> > > > > > > > "id":{  
> > > > > > > > "type":"string",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > "store":"yes"  
> > > > > > > > },  
> > > > > > > > "pkgs":{  
> > > > > > > > "type":"string",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "lcs":{  
> > > > > > > > "type":"string",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "from":{  
> > > > > > > > "type":"long",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "to":{  
> > > > > > > > "type":"long",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "status":{  
> > > > > > > > "type":"integer",  
> > > > > > > > "index":"not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "nets" : {  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "androidAPILevels":{  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "hMin":{  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "hMax":{  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "wMin":{  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "wMax":{  
> > > > > > > > "type" : "integer",  
> > > > > > > > "index" : "not\_analyzed",  
> > > > > > > > "store":"no"  
> > > > > > > > },  
> > > > > > > > "models":{  
> > > > > > > > "type" : "string",  
> > > > > > > > "index": "analyzed",  
> > > > > > > > "index\_analyzer":"standardAnalyzer",  
> > > > > > > > "search\_analyzer":"standardAnalyzer",  
> > > > > > > > "store":"no"  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }
> > 
> > > > > > > > }

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 16, 2011, 12:53pm UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/10 "2011-12-16T12:53:50Z")

</div>

sorry, i misunderstood.  
i checked ulimit -l and it shows that the max locked memory is only  
32k.

Today, i have changed from es to pure lucene memory index(with  
RAMDirectory) and the speed is much faster and only 500m+ memory is  
ued. Current the system can response 2000 requests per second.

I will set ulimit -l and test es again, thanks clint

On Dec 16, 5:43 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:

> On Thu, 2011-12-15 at 18:29 -0800, Weiwei Wang wrote:
> 
> > [2011-12-15 17:05:54,090][INFO][bootstrap]  
> > max\_open\_files[65510]
> 
> ulimit -l is to do with locking memory, not open files. that is ulimit  
> -n
> 
> clint
> 
> > i think it's enough.
> 
> > I've already use es for another project with more than 2620000  
> > documents(10g+) but with low pressure. However, when i reindex all the  
> > documents i find es eat so much memory (5g+), so i set -Xms8g -Xmx8g  
> > for that project. I doubt that es has memory leak.
> 
> > On Dec 15, 11:42 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:
> > > 
> > > > i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> > > > files=true -Dbootstrap.mlockall=true -p es.pid
> 
> > > you don't mention whether you are setting: ulimit -l unlimited
> 
> > > without that, mlockall won't work.
> 
> > > clint
> 
> > > > On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > > 
> > > > > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > > > > 
> > > > > > only one elasticsearch instance with one shard, 0 replica. I tested  
> > > > > > another time with 4g maximum memeory, after one hour, the problem  
> > > > > > occurs again. I stop jmeter and use top to monitor the es process and  
> > > > > > found the memory usage stays on 4.2g and cpu load jump from 20+% to  
> > > > > > 300%. I was considering using es in my program for a high load web  
> > > > > > service project, now i have to consider using pure lucene.
> 
> > > > > Are you disabling swap, ie configuring bootstrap.mlockall, ulimit -l,  
> > > > > and ES\_MIN/MAX\_MEM correctly
> 
> > > > > There are numerous emails explaining how to do this in this list, so a  
> > > > > quick search should find some
> 
> > > > > clint
> 
> > > > > > my es config is shown as:  
> > > > > > {  
> > > > > > "cluster":{  
> > > > > > "name":"es-cluster"  
> > > > > > },  
> > > > > > "gateway":{  
> > > > > > "recover\_after\_nodes": 1,  
> > > > > > "recover\_after\_time": "30s",  
> > > > > > "expected\_nodes": 2  
> > > > > > },  
> > > > > > "network":  
> > > > > > {  
> > > > > > "host":"0.0.0.0",  
> > > > > > "tcp":{  
> > > > > > "keep\_alive":true,  
> > > > > > "send\_buffer\_size":"50m",  
> > > > > > "receive\_buffer\_size":"50m"  
> > > > > > }  
> > > > > > },  
> > > > > > "transport":{  
> > > > > > "tcp":{  
> > > > > > "port":"9350-9400",  
> > > > > > "keep\_alive":true,  
> > > > > > "send\_buffer\_size":"20m",  
> > > > > > "receive\_buffer\_size":"20m",  
> > > > > > "connect\_timeout":"5s"  
> > > > > > }  
> > > > > > },  
> > > > > > "http":{  
> > > > > > "port":"9250-9300"  
> > > > > > },  
> > > > > > "index" : {  
> > > > > > "store":{  
> > > > > > "cache":{  
> > > > > > "memory":{  
> > > > > > "small\_buffer\_size":"32mb",  
> > > > > > "large\_buffer\_size":"64mb",  
> > > > > > "small\_cache\_size":"512mb",  
> > > > > > "large\_cache\_size":"1g"  
> > > > > > }  
> > > > > > }  
> > > > > > },  
> > > > > > "search":{  
> > > > > > "slowlog":{  
> > > > > > "threshold":{  
> > > > > > "query":{  
> > > > > > "warn":"1s",  
> > > > > > "info":"500ms",  
> > > > > > "debug":"200ms",  
> > > > > > "trace":"50ms"  
> > > > > > },  
> > > > > > "fetch":{  
> > > > > > "warn":"100ms",  
> > > > > > "info":"80ms",  
> > > > > > "debug":"50ms",  
> > > > > > "trace":"20ms"  
> > > > > > }  
> > > > > > }  
> > > > > > }  
> > > > > > },  
> > > > > > "number\_of\_shards":2,  
> > > > > > "number\_of\_replicas":1,  
> > > > > > "refresh\_interval":"1s",  
> > > > > > "term\_index\_interval":64,  
> > > > > > "analysis" : {  
> > > > > > "analyzer" : {  
> > > > > > "nGramAnalyzer":{  
> > > > > > "type":"custom",  
> > > > > > "tokenizer":"standard",  
> > > > > > "filter":  
> > > > > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > > > > },  
> > > > > > "standardAnalyzer":{  
> > > > > > "type":"custom",  
> > > > > > "tokenizer":"standard",  
> > > > > > "filter":  
> > > > > > ["standard","lowercase","englishSnowball"]  
> > > > > > }  
> > > > > > },  
> > > > > > "filter":{  
> > > > > > "nGramFilter":{  
> > > > > > "type":"nGram",  
> > > > > > "min\_gram":1,  
> > > > > > "max\_gram":64  
> > > > > > },  
> > > > > > "edgeNGramFilter":{  
> > > > > > "type":"edgeNGram",  
> > > > > > "min\_gram":1,  
> > > > > > "max\_gram":64,  
> > > > > > "side":"front"  
> > > > > > },  
> > > > > > "englishSnowball":{  
> > > > > > "type":"snowball",  
> > > > > > "language":"English"  
> > > > > > }  
> > > > > > }  
> > > > > > }  
> > > > > > }  
> > > > > > }
> 
> > > > > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote:
> > > > > > 
> > > > > > > you can try to give it a lot less memory (e.g. \<50MB ?) and then  
> > > > > > > garbage collector will be called more often.
> 
> > > > > > > not sure why ES will take so much time after an hour. how many shards  
> > > > > > > do you have + how many CPUs? also try to monitor with ES-DESK or  
> > > > > > > jvisualvm to see the real RAM usage and CPU load while testing.
> 
> > > > > > > Peter.
> 
> > > > > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > > > > another problem is though there is only 200 documents and 100k data  
> > > > > > > > storage, the memory use under pressure is more than 2g(with top under  
> > > > > > > > linux). I don't know why so much memory is needed.
> 
> > > > > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > > > > > I test my program with high pressure of 1500 request per second and  
> > > > > > > > > each request need to do a es query(i use MatchAllQuery and a bunch of  
> > > > > > > > > filters as filters can be cached). The size of the index is 106.7kb  
> > > > > > > > > with 200 documents. I start my es with paramters: bin/elasticsearch -  
> > > > > > > > > Xms2g -Xmx2g -Des.max-open-files=true -Dbootstrap.mlockall=true
> 
> > > > > > > > > After a hour, es begins to become slow for query and from log i can  
> > > > > > > > > see lots of slow query, i paste some logs below and wish your help:
> 
> > > > > > > > > [2011-12-15 13:49:22,050][WARN][index.search.slowlog.query]  
> > > > > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2], source[{"from":  
> > > > > > > > > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":  
> > > > > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":  
> > > > > > > > > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":  
> > > > > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":  
> > > > > > > > > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":  
> > > > > > > > > {"range":{"to":{"from":  
> > > > > > > > > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must ":  
> > > > > > > > > {"range":{"hMax":{"from":  
> > > > > > > > > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > > > {"range":{"hMin":{"from":null,"to":  
> > > > > > > > > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":  
> > > > > > > > > {"wMax":{"from":  
> > > > > > > > > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":  
> > > > > > > > > {"range":{"wMin":{"from":null,"to":  
> > > > > > > > > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields" :"id"}  
> > > > > > > > > extra\_source,
> 
> > > > > > > > > the mapping are:  
> > > > > > > > > {  
> > > > > > > > > "test":{  
> > > > > > > > > "\_all" : {  
> > > > > > > > > "enabled" : false  
> > > > > > > > > },  
> > > > > > > > > "properties" : {  
> > > > > > > > > "id":{  
> > > > > > > > > "type":"string",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > "store":"yes"  
> > > > > > > > > },  
> > > > > > > > > "pkgs":{  
> > > > > > > > > "type":"string",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > "store":"no"  
> > > > > > > > > },  
> > > > > > > > > "lcs":{  
> > > > > > > > > "type":"string",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > "store":"no"  
> > > > > > > > > },  
> > > > > > > > > "from":{  
> > > > > > > > > "type":"long",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "store":"no"  
> > > > > > > > > },  
> > > > > > > > > "to":{  
> > > > > > > > > "type":"long",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "store":"no"  
> > > > > > > > > },  
> > > > > > > > > "status":{  
> > > > > > > > > "type":"integer",  
> > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > "store":"no"
> 
> ...
> 
> read more »

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [December 16, 2011, 5:45pm UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/11 "2011-12-16T17:45:29Z")

</div>

First of all, since you allocated 2gb to ES, it will use it (min/max),  
thats why you see it takes 2gb. Second, where do you run jmeter? Is it on  
the same box as the elasticsearch instance? If you want, you can dropbox /  
shard the data directory of an already indexed data and the jmx jmeter  
file, I can have a look.

On Fri, Dec 16, 2011 at 2:53 PM, Weiwei Wang [ww.wang.cs@gmail.com](mailto:ww.wang.cs@gmail.com) wrote:

> sorry, i misunderstood.  
> i checked ulimit -l and it shows that the max locked memory is only  
> 32k.
> 
> Today, i have changed from es to pure lucene memory index(with  
> RAMDirectory) and the speed is much faster and only 500m+ memory is  
> ued. Current the system can response 2000 requests per second.
> 
> I will set ulimit -l and test es again, thanks clint
> 
> On Dec 16, 5:43 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> 
> > On Thu, 2011-12-15 at 18:29 -0800, Weiwei Wang wrote:
> > 
> > > [2011-12-15 17:05:54,090][INFO][bootstrap]  
> > > max\_open\_files[65510]
> > 
> > ulimit -l is to do with locking memory, not open files. that is ulimit  
> > -n
> > 
> > clint
> > 
> > > i think it's enough.
> > 
> > > I've already use es for another project with more than 2620000  
> > > documents(10g+) but with low pressure. However, when i reindex all the  
> > > documents i find es eat so much memory (5g+), so i set -Xms8g -Xmx8g  
> > > for that project. I doubt that es has memory leak.
> > 
> > > On Dec 15, 11:42 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > 
> > > > On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:
> > > > 
> > > > > i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> > > > > files=true -Dbootstrap.mlockall=true -p es.pid
> > 
> > > > you don't mention whether you are setting: ulimit -l unlimited
> > 
> > > > without that, mlockall won't work.
> > 
> > > > clint
> > 
> > > > > On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > > > 
> > > > > > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > > > > > 
> > > > > > > only one elasticsearch instance with one shard, 0 replica. I  
> > > > > > > tested  
> > > > > > > another time with 4g maximum memeory, after one hour, the  
> > > > > > > problem  
> > > > > > > occurs again. I stop jmeter and use top to monitor the es  
> > > > > > > process and  
> > > > > > > found the memory usage stays on 4.2g and cpu load jump from  
> > > > > > > 20+% to  
> > > > > > > 300%. I was considering using es in my program for a high load  
> > > > > > > web  
> > > > > > > service project, now i have to consider using pure lucene.
> > 
> > > > > > Are you disabling swap, ie configuring bootstrap.mlockall,  
> > > > > > ulimit -l,  
> > > > > > and ES\_MIN/MAX\_MEM correctly
> > 
> > > > > > There are numerous emails explaining how to do this in this  
> > > > > > list, so a  
> > > > > > quick search should find some
> > 
> > > > > > clint
> > 
> > > > > > > my es config is shown as:  
> > > > > > > {  
> > > > > > > "cluster":{  
> > > > > > > "name":"es-cluster"  
> > > > > > > },  
> > > > > > > "gateway":{  
> > > > > > > "recover\_after\_nodes": 1,  
> > > > > > > "recover\_after\_time": "30s",  
> > > > > > > "expected\_nodes": 2  
> > > > > > > },  
> > > > > > > "network":  
> > > > > > > {  
> > > > > > > "host":"0.0.0.0",  
> > > > > > > "tcp":{  
> > > > > > > "keep\_alive":true,  
> > > > > > > "send\_buffer\_size":"50m",  
> > > > > > > "receive\_buffer\_size":"50m"  
> > > > > > > }  
> > > > > > > },  
> > > > > > > "transport":{  
> > > > > > > "tcp":{  
> > > > > > > "port":"9350-9400",  
> > > > > > > "keep\_alive":true,  
> > > > > > > "send\_buffer\_size":"20m",  
> > > > > > > "receive\_buffer\_size":"20m",  
> > > > > > > "connect\_timeout":"5s"  
> > > > > > > }  
> > > > > > > },  
> > > > > > > "http":{  
> > > > > > > "port":"9250-9300"  
> > > > > > > },  
> > > > > > > "index" : {  
> > > > > > > "store":{  
> > > > > > > "cache":{  
> > > > > > > "memory":{  
> > > > > > > "small\_buffer\_size":"32mb",  
> > > > > > > "large\_buffer\_size":"64mb",  
> > > > > > > "small\_cache\_size":"512mb",  
> > > > > > > "large\_cache\_size":"1g"  
> > > > > > > }  
> > > > > > > }  
> > > > > > > },  
> > > > > > > "search":{  
> > > > > > > "slowlog":{  
> > > > > > > "threshold":{  
> > > > > > > "query":{  
> > > > > > > "warn":"1s",  
> > > > > > > "info":"500ms",  
> > > > > > > "debug":"200ms",  
> > > > > > > "trace":"50ms"  
> > > > > > > },  
> > > > > > > "fetch":{  
> > > > > > > "warn":"100ms",  
> > > > > > > "info":"80ms",  
> > > > > > > "debug":"50ms",  
> > > > > > > "trace":"20ms"  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }  
> > > > > > > },  
> > > > > > > "number\_of\_shards":2,  
> > > > > > > "number\_of\_replicas":1,  
> > > > > > > "refresh\_interval":"1s",  
> > > > > > > "term\_index\_interval":64,  
> > > > > > > "analysis" : {  
> > > > > > > "analyzer" : {  
> > > > > > > "nGramAnalyzer":{  
> > > > > > > "type":"custom",  
> > > > > > > "tokenizer":"standard",  
> > > > > > > "filter":  
> > > > > > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > > > > > },  
> > > > > > > "standardAnalyzer":{  
> > > > > > > "type":"custom",  
> > > > > > > "tokenizer":"standard",  
> > > > > > > "filter":  
> > > > > > > ["standard","lowercase","englishSnowball"]  
> > > > > > > }  
> > > > > > > },  
> > > > > > > "filter":{  
> > > > > > > "nGramFilter":{  
> > > > > > > "type":"nGram",  
> > > > > > > "min\_gram":1,  
> > > > > > > "max\_gram":64  
> > > > > > > },  
> > > > > > > "edgeNGramFilter":{  
> > > > > > > "type":"edgeNGram",  
> > > > > > > "min\_gram":1,  
> > > > > > > "max\_gram":64,  
> > > > > > > "side":"front"  
> > > > > > > },  
> > > > > > > "englishSnowball":{  
> > > > > > > "type":"snowball",  
> > > > > > > "language":"English"  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }  
> > > > > > > }
> > 
> > > > > > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com)  
> > > > > > > wrote:
> > > > > > > 
> > > > > > > > you can try to give it a lot less memory (e.g. \<50MB ?) and  
> > > > > > > > then  
> > > > > > > > garbage collector will be called more often.
> > 
> > > > > > > > not sure why ES will take so much time after an hour. how  
> > > > > > > > many shards  
> > > > > > > > do you have + how many CPUs? also try to monitor with  
> > > > > > > > ES-DESK or  
> > > > > > > > jvisualvm to see the real RAM usage and CPU load while  
> > > > > > > > testing.
> > 
> > > > > > > > Peter.
> > 
> > > > > > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> > 
> > > > > > > > > another problem is though there is only 200 documents and  
> > > > > > > > > 100k data  
> > > > > > > > > storage, the memory use under pressure is more than  
> > > > > > > > > 2g(with top under  
> > > > > > > > > linux). I don't know why so much memory is needed.
> > 
> > > > > > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com)  
> > > > > > > > > wrote:
> > 
> > > > > > > > > > I test my program with high pressure of 1500 request  
> > > > > > > > > > per second and  
> > > > > > > > > > each request need to do a es query(i use MatchAllQuery  
> > > > > > > > > > and a bunch of  
> > > > > > > > > > filters as filters can be cached). The size of the index  
> > > > > > > > > > is 106.7kb  
> > > > > > > > > > with 200 documents. I start my es with paramters:  
> > > > > > > > > > bin/elasticsearch -  
> > > > > > > > > > Xms2g -Xmx2g -Des.max-open-files=true  
> > > > > > > > > > -Dbootstrap.mlockall=true
> > 
> > > > > > > > > > After a hour, es begins to become slow for query and  
> > > > > > > > > > from log i can  
> > > > > > > > > > see lots of slow query, i paste some logs below and wish  
> > > > > > > > > > your help:
> > 
> > > > > > > > > > [2011-12-15 13:49:22,050][WARN  
> > > > > > > > > > ][index.search.slowlog.query]  
> > > > > > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2],  
> > > > > > > > > > source[{"from":
> 
> 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":
> 
> > > > > > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":
> 
> {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":
> 
> > > > > > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":
> 
> 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":
> 
> > > > > > > > > > {"range":{"to":{"from":
> 
> 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must  
> ":
> 
> > > > > > > > > > {"range":{"hMax":{"from":
> 
> 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":
> 
> > > > > > > > > > {"range":{"hMin":{"from":null,"to":
> 
> 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":
> 
> > > > > > > > > > {"wMax":{"from":
> 
> 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":
> 
> > > > > > > > > > {"range":{"wMin":{"from":null,"to":
> 
> 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields"  
> :"id"}
> 
> > > > > > > > > > extra\_source,
> > 
> > > > > > > > > > the mapping are:  
> > > > > > > > > > {  
> > > > > > > > > > "test":{  
> > > > > > > > > > "\_all" : {  
> > > > > > > > > > "enabled" : false  
> > > > > > > > > > },  
> > > > > > > > > > "properties" : {  
> > > > > > > > > > "id":{  
> > > > > > > > > > "type":"string",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > > "store":"yes"  
> > > > > > > > > > },  
> > > > > > > > > > "pkgs":{  
> > > > > > > > > > "type":"string",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > > "store":"no"  
> > > > > > > > > > },  
> > > > > > > > > > "lcs":{  
> > > > > > > > > > "type":"string",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "search\_analyzer":"keyword",  
> > > > > > > > > > "store":"no"  
> > > > > > > > > > },  
> > > > > > > > > > "from":{  
> > > > > > > > > > "type":"long",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "store":"no"  
> > > > > > > > > > },  
> > > > > > > > > > "to":{  
> > > > > > > > > > "type":"long",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "store":"no"  
> > > > > > > > > > },  
> > > > > > > > > > "status":{  
> > > > > > > > > > "type":"integer",  
> > > > > > > > > > "index":"not\_analyzed",  
> > > > > > > > > > "store":"no"
> > 
> > ...
> > 
> > read more »

---

<div class="post-metadata">

**Author:** ![Weiwei\_Wang](https://avatars.discourse-cdn.com/v4/letter/w/b19c9b/32.png) [@Weiwei\_Wang](https://discuss.elastic.co/u/Weiwei_Wang)\
**Post date:** [December 17, 2011, 3:19am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/12 "2011-12-17T03:19:37Z")

</div>

it's ok for es to use all the memory, but the phenomenon is that the  
memory is increasing slowly in the pressure progress and finally  
reaches the limit, after that slow query log comes. But I have only  
200 documents and 100k+ index storage. I change from es to Lucene  
memory index and the system can run under high load for a long  
time(more than 2 hours) with 500m+ memory.

i have sent the data and jmeter script to your gmail,shay

thanks~

On Dec 17, 1:45 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> First of all, since you allocated 2gb to ES, it will use it (min/max),  
> thats why you see it takes 2gb. Second, where do you run jmeter? Is it on  
> the same box as the elasticsearch instance? If you want, you can dropbox /  
> shard the data directory of an already indexed data and the jmx jmeter  
> file, I can have a look.
> 
> On Fri, Dec 16, 2011 at 2:53 PM, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > sorry, i misunderstood.  
> > i checked ulimit -l and it shows that the max locked memory is only  
> > 32k.
> 
> > Today, i have changed from es to pure lucene memory index(with  
> > RAMDirectory) and the speed is much faster and only 500m+ memory is  
> > ued. Current the system can response 2000 requests per second.
> 
> > I will set ulimit -l and test es again, thanks clint
> 
> > On Dec 16, 5:43 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > 
> > > On Thu, 2011-12-15 at 18:29 -0800, Weiwei Wang wrote:
> > > 
> > > > [2011-12-15 17:05:54,090][INFO][bootstrap]  
> > > > max\_open\_files[65510]
> 
> > > ulimit -l is to do with locking memory, not open files. that is ulimit  
> > > -n
> 
> > > clint
> 
> > > > i think it's enough.
> 
> > > > I've already use es for another project with more than 2620000  
> > > > documents(10g+) but with low pressure. However, when i reindex all the  
> > > > documents i find es eat so much memory (5g+), so i set -Xms8g -Xmx8g  
> > > > for that project. I doubt that es has memory leak.
> 
> > > > On Dec 15, 11:42 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > > 
> > > > > On Thu, 2011-12-15 at 07:24 -0800, Weiwei Wang wrote:
> > > > > 
> > > > > > i start es with bin/elasticsearch -Xms2g -Xmx2g -Des.max-open-  
> > > > > > files=true -Dbootstrap.mlockall=true -p es.pid
> 
> > > > > you don't mention whether you are setting: ulimit -l unlimited
> 
> > > > > without that, mlockall won't work.
> 
> > > > > clint
> 
> > > > > > On Dec 15, 8:00 pm, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote:
> > > > > > 
> > > > > > > On Thu, 2011-12-15 at 01:03 -0800, Weiwei Wang wrote:
> > > > > > > 
> > > > > > > > only one elasticsearch instance with one shard, 0 replica. I  
> > > > > > > > tested  
> > > > > > > > another time with 4g maximum memeory, after one hour, the  
> > > > > > > > problem  
> > > > > > > > occurs again. I stop jmeter and use top to monitor the es  
> > > > > > > > process and  
> > > > > > > > found the memory usage stays on 4.2g and cpu load jump from  
> > > > > > > > 20+% to  
> > > > > > > > 300%. I was considering using es in my program for a high load  
> > > > > > > > web  
> > > > > > > > service project, now i have to consider using pure lucene.
> 
> > > > > > > Are you disabling swap, ie configuring bootstrap.mlockall,  
> > > > > > > ulimit -l,  
> > > > > > > and ES\_MIN/MAX\_MEM correctly
> 
> > > > > > > There are numerous emails explaining how to do this in this  
> > > > > > > list, so a  
> > > > > > > quick search should find some
> 
> > > > > > > clint
> 
> > > > > > > > my es config is shown as:  
> > > > > > > > {  
> > > > > > > > "cluster":{  
> > > > > > > > "name":"es-cluster"  
> > > > > > > > },  
> > > > > > > > "gateway":{  
> > > > > > > > "recover\_after\_nodes": 1,  
> > > > > > > > "recover\_after\_time": "30s",  
> > > > > > > > "expected\_nodes": 2  
> > > > > > > > },  
> > > > > > > > "network":  
> > > > > > > > {  
> > > > > > > > "host":"0.0.0.0",  
> > > > > > > > "tcp":{  
> > > > > > > > "keep\_alive":true,  
> > > > > > > > "send\_buffer\_size":"50m",  
> > > > > > > > "receive\_buffer\_size":"50m"  
> > > > > > > > }  
> > > > > > > > },  
> > > > > > > > "transport":{  
> > > > > > > > "tcp":{  
> > > > > > > > "port":"9350-9400",  
> > > > > > > > "keep\_alive":true,  
> > > > > > > > "send\_buffer\_size":"20m",  
> > > > > > > > "receive\_buffer\_size":"20m",  
> > > > > > > > "connect\_timeout":"5s"  
> > > > > > > > }  
> > > > > > > > },  
> > > > > > > > "http":{  
> > > > > > > > "port":"9250-9300"  
> > > > > > > > },  
> > > > > > > > "index" : {  
> > > > > > > > "store":{  
> > > > > > > > "cache":{  
> > > > > > > > "memory":{  
> > > > > > > > "small\_buffer\_size":"32mb",  
> > > > > > > > "large\_buffer\_size":"64mb",  
> > > > > > > > "small\_cache\_size":"512mb",  
> > > > > > > > "large\_cache\_size":"1g"  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > },  
> > > > > > > > "search":{  
> > > > > > > > "slowlog":{  
> > > > > > > > "threshold":{  
> > > > > > > > "query":{  
> > > > > > > > "warn":"1s",  
> > > > > > > > "info":"500ms",  
> > > > > > > > "debug":"200ms",  
> > > > > > > > "trace":"50ms"  
> > > > > > > > },  
> > > > > > > > "fetch":{  
> > > > > > > > "warn":"100ms",  
> > > > > > > > "info":"80ms",  
> > > > > > > > "debug":"50ms",  
> > > > > > > > "trace":"20ms"  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > },  
> > > > > > > > "number\_of\_shards":2,  
> > > > > > > > "number\_of\_replicas":1,  
> > > > > > > > "refresh\_interval":"1s",  
> > > > > > > > "term\_index\_interval":64,  
> > > > > > > > "analysis" : {  
> > > > > > > > "analyzer" : {  
> > > > > > > > "nGramAnalyzer":{  
> > > > > > > > "type":"custom",  
> > > > > > > > "tokenizer":"standard",  
> > > > > > > > "filter":  
> > > > > > > > ["standard","lowercase","englishSnowball","nGramFilter"]  
> > > > > > > > },  
> > > > > > > > "standardAnalyzer":{  
> > > > > > > > "type":"custom",  
> > > > > > > > "tokenizer":"standard",  
> > > > > > > > "filter":  
> > > > > > > > ["standard","lowercase","englishSnowball"]  
> > > > > > > > }  
> > > > > > > > },  
> > > > > > > > "filter":{  
> > > > > > > > "nGramFilter":{  
> > > > > > > > "type":"nGram",  
> > > > > > > > "min\_gram":1,  
> > > > > > > > "max\_gram":64  
> > > > > > > > },  
> > > > > > > > "edgeNGramFilter":{  
> > > > > > > > "type":"edgeNGram",  
> > > > > > > > "min\_gram":1,  
> > > > > > > > "max\_gram":64,  
> > > > > > > > "side":"front"  
> > > > > > > > },  
> > > > > > > > "englishSnowball":{  
> > > > > > > > "type":"snowball",  
> > > > > > > > "language":"English"  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }  
> > > > > > > > }
> 
> > > > > > > > On Dec 15, 4:23 pm, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com)  
> > > > > > > > wrote:
> > > > > > > > 
> > > > > > > > > you can try to give it a lot less memory (e.g. \<50MB ?) and  
> > > > > > > > > then  
> > > > > > > > > garbage collector will be called more often.
> 
> > > > > > > > > not sure why ES will take so much time after an hour. how  
> > > > > > > > > many shards  
> > > > > > > > > do you have + how many CPUs? also try to monitor with  
> > > > > > > > > ES-DESK or  
> > > > > > > > > jvisualvm to see the real RAM usage and CPU load while  
> > > > > > > > > testing.
> 
> > > > > > > > > Peter.
> 
> > > > > > > > > On 15 Dez., 06:59, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com) wrote:
> 
> > > > > > > > > > another problem is though there is only 200 documents and  
> > > > > > > > > > 100k data  
> > > > > > > > > > storage, the memory use under pressure is more than  
> > > > > > > > > > 2g(with top under  
> > > > > > > > > > linux). I don't know why so much memory is needed.
> 
> > > > > > > > > > On Dec 15, 1:55 pm, Weiwei Wang [ww.wang...@gmail.com](mailto:ww.wang...@gmail.com)  
> > > > > > > > > > wrote:
> 
> > > > > > > > > > > I test my program with high pressure of 1500 request  
> > > > > > > > > > > per second and  
> > > > > > > > > > > each request need to do a es query(i use MatchAllQuery  
> > > > > > > > > > > and a bunch of  
> > > > > > > > > > > filters as filters can be cached). The size of the index  
> > > > > > > > > > > is 106.7kb  
> > > > > > > > > > > with 200 documents. I start my es with paramters:  
> > > > > > > > > > > bin/elasticsearch -  
> > > > > > > > > > > Xms2g -Xmx2g -Des.max-open-files=true  
> > > > > > > > > > > -Dbootstrap.mlockall=true
> 
> > > > > > > > > > > After a hour, es begins to become slow for query and  
> > > > > > > > > > > from log i can  
> > > > > > > > > > > see lots of slow query, i paste some logs below and wish  
> > > > > > > > > > > your help:
> 
> > > > > > > > > > > [2011-12-15 13:49:22,050][WARN  
> > > > > > > > > > > ][index.search.slowlog.query]  
> > > > > > > > > > > [Nefarius] [dianxin][1] took[8.3s], took\_millis[8329],  
> > > > > > > > > > > search\_type[QUERY\_THEN\_FETCH], total\_shards[2],  
> > > > > > > > > > > source[{"from":
> 
> > 0,"size":1,"query":{"match\_all":{}},"filter":{"bool":{"must":{"term":
> > 
> > > > > > > > > > > {"pkgs":"recommendation.test.pkg.3"}},"must":{"term":  
> > > > > > > > > > > {"lcs":"recommendation.test.lc.3"}},"must":{"term":
> 
> > {"nets":"1"}},"must":{"term":{"androidAPILevels":"9"}},"must":{"term":
> > 
> > > > > > > > > > > {"status":1}},"must":{"range":{"from":{"from":null,"to":
> 
> > 1323928153712,"include\_lower":true,"include\_upper":false}}},"must":
> > 
> > > > > > > > > > > {"range":{"to":{"from":
> 
> > 1323928153712,"to":null,"include\_lower":false,"include\_upper":true}}},"must  
> > ":
> > 
> > > > > > > > > > > {"range":{"hMax":{"from":
> 
> > 800,"to":null,"include\_lower":true,"include\_upper":true}}},"must":
> > 
> > > > > > > > > > > {"range":{"hMin":{"from":null,"to":
> 
> > 800,"include\_lower":true,"include\_upper":true}}},"must":{"range":
> > 
> > > > > > > > > > > {"wMax":{"from":
> 
> > 480,"to":null,"include\_lower":true,"include\_upper":true}}},"must":
> > 
> > > > > > > > > > > {"range":{"wMin":{"from":null,"to":
> 
> > 480,"include\_lower":true,"include\_upper":true}}}}},"explain":false,"fields"  
> > :"id"}
> > 
> > > > > > > > > > > extra\_source,
> 
> > > > > > > > > > > the mapping are:  
> > > > > > > > > > > {  
> > > > > > > > > > > "test":{
> 
> ...
> 
> read more »

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/many-slow-query-with-high-load-after-a-hour/6162/13 "2017-07-06T03:45:02Z")

</div>