# CPU and memory usage suddenly spiral out of control **URL:** \ **Category:** Elasticsearch\ **Created:** [June 14, 2011, 7:28pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621 "2011-06-14T19:28:29Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 14, 2011, 7:28pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/1 "2011-06-14T19:28:29Z")
Any ideas of why a query might work fine, and then suddenly spike out of control? I have the logs, running at DEBUG level, from an example incident. The same query gets executed, and on the 10th or so run, the memory starts piling up, and top shows the process taking anywhere from 100 to 177% of the CPU while this is happening. > I can get it to occur pretty easily after I restart the server, but only on my EC2 instance. On my local machine, it's fine. The query is a dis\_max of 6 text/phrase queries on different fields, on an index of about 17,000 documents, where each document ranges from a few kilobytes to a few megabytes (not many of those). The data directory takes up 2.4GB of space. Sorry for all the emails - I guess today's my day for questions! -- Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 14, 2011, 8:25pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/2 "2011-06-14T20:25:26Z")
Some typical things to check: 1. Make sure the ES (java) process is not swapping. You can force it not to swap by setting bootstrap.mlockall to true. 2. Which operating system are you running? Ubuntu 10.04 is known to have problems on ec2. 3. Are you indexing while you do the search? 4. Where do you store the data directory? Is it on local drive or EBS? Maybe EBS is misbehaving? On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > Any ideas of why a query might work fine, and then suddenly spike out of control? I have the logs, running at DEBUG level, from an example incident. The same query gets executed, and on the 10th or so run, the memory starts piling up, and top shows the process taking anywhere from 100 to 177% of the CPU while this is happening. > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > I can get it to occur pretty easily after I restart the server, but only on my EC2 instance. On my local machine, it's fine. > > The query is a dis\_max of 6 text/phrase queries on different fields, on an index of about 17,000 documents, where each document ranges from a few kilobytes to a few megabytes (not many of those). The data directory takes up 2.4GB of space. > > Sorry for all the emails - I guess today's my day for questions! > > -- Eric ---
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 14, 2011, 10:59pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/3 "2011-06-14T22:59:57Z")
1. I'll try that. 2. It's Ubuntu 10.10. Still known problems? 3. No, indexing is done in bulk once a night (for now). 4. The data directory is on an EBS, but it doesn't appear to be misbehaving. I'll investigate a bit more on that front. Thanks, Shay, -- Eric On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Some typical things to check: > > 1. Make sure the ES (java) process is not swapping. You can force it not to > swap by setting bootstrap.mlockall to true. > 2. Which operating system are you running? Ubuntu 10.04 is known to have > problems on ec2. > 3. Are you indexing while you do the search? > 4. Where do you store the data directory? Is it on local drive or EBS? > Maybe EBS is misbehaving? > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > Any ideas of why a query might work fine, and then suddenly spike out of > control? I have the logs, running at DEBUG level, from an example incident. > The same query gets executed, and on the 10th or so run, the memory starts > piling up, and top shows the process taking anywhere from 100 to 177% of the > CPU while this is happening. > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > I can get it to occur pretty easily after I restart the server, but only on > my EC2 instance. On my local machine, it's fine. > > The query is a dis\_max of 6 text/phrase queries on different fields, on an > index of about 17,000 documents, where each document ranges from a few > kilobytes to a few megabytes (not many of those). The data directory takes > up 2.4GB of space. > > Sorry for all the emails - I guess today's my day for questions! > > -- Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 14, 2011, 11:04pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/4 "2011-06-14T23:04:34Z")
Few more questions: 1. Which ec2 instance type are you using? The smaller you get, the more likely they are hurt by "virtualization noise", also, make sure to use ones with high IO. 2. Which elasticsearch version are you using? I am not familiar with problems on 10.10, though I heard bad stories about it as well (from people using redis) on ec2, but did not encounter it with elasticsearch. On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > 1. I'll try that. > 2. It's Ubuntu 10.10. Still known problems? > 3. No, indexing is done in bulk once a night (for now). > 4. The data directory is on an EBS, but it doesn't appear to be misbehaving. I'll investigate a bit more on that front. > > Thanks, Shay, > > -- Eric > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > Some typical things to check: > > > > 1. Make sure the ES (java) process is not swapping. You can force it not to swap by setting bootstrap.mlockall to true. > > 2. Which operating system are you running? Ubuntu 10.04 is known to have problems on ec2. > > 3. Are you indexing while you do the search? > > 4. Where do you store the data directory? Is it on local drive or EBS? Maybe EBS is misbehaving? > > > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > > > > Any ideas of why a query might work fine, and then suddenly spike out of control? I have the logs, running at DEBUG level, from an example incident. The same query gets executed, and on the 10th or so run, the memory starts piling up, and top shows the process taking anywhere from 100 to 177% of the CPU while this is happening. > > > > > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > > > > > I can get it to occur pretty easily after I restart the server, but only on my EC2 instance. On my local machine, it's fine. > > > > > > The query is a dis\_max of 6 text/phrase queries on different fields, on an index of about 17,000 documents, where each document ranges from a few kilobytes to a few megabytes (not many of those). The data directory takes up 2.4GB of space. > > > > > > Sorry for all the emails - I guess today's my day for questions! > > > > > > -- Eric ---
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 15, 2011, 1:31am UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/5 "2011-06-15T01:31:05Z")
It's an m1.large, which has high I/O. I'm using the latest version, elasticsearch v0.16.2. I'll try tomorrow morning to see if preventing it from swapping fixes it. -- Eric On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Few more questions: > > 1. Which ec2 instance type are you using? The smaller you get, the more > likely they are hurt by "virtualization noise", also, make sure to use ones > with high IO. > 2. Which elasticsearch version are you using? > > I am not familiar with problems on 10.10, though I heard bad stories about > it as well (from people using redis) on ec2, but did not encounter it with > elasticsearch. > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > 1. I'll try that. > 2. It's Ubuntu 10.10. Still known problems? > 3. No, indexing is done in bulk once a night (for now). > 4. The data directory is on an EBS, but it doesn't appear to be > misbehaving. I'll investigate a bit more on that front. > > Thanks, Shay, > > -- Eric > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Some typical things to check: > > 1. Make sure the ES (java) process is not swapping. You can force it not to > swap by setting bootstrap.mlockall to true. > 2. Which operating system are you running? Ubuntu 10.04 is known to have > problems on ec2. > 3. Are you indexing while you do the search? > 4. Where do you store the data directory? Is it on local drive or EBS? > Maybe EBS is misbehaving? > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > Any ideas of why a query might work fine, and then suddenly spike out of > control? I have the logs, running at DEBUG level, from an example incident. > The same query gets executed, and on the 10th or so run, the memory starts > piling up, and top shows the process taking anywhere from 100 to 177% of the > CPU while this is happening. > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > I can get it to occur pretty easily after I restart the server, but only on > my EC2 instance. On my local machine, it's fine. > > The query is a dis\_max of 6 text/phrase queries on different fields, on an > index of about 17,000 documents, where each document ranges from a few > kilobytes to a few megabytes (not many of those). The data directory takes > up 2.4GB of space. > > Sorry for all the emails - I guess today's my day for questions! > > -- Eric ---
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 15, 2011, 11:47pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/6 "2011-06-15T23:47:21Z")
OK, I've got a lot more info about the problem. I've now isolated it to a query, not a machine. EC2 isn't the problem, as I first thought it was. I can trigger Elasticsearch to go into a death spiral (where it spikes the CPU and stays there until kill -9'd) the first time this query is run on this set of documents. The query is a dis\_max on a set of 6 "text" queries of type "phrase", each for the term "cap and trade". Highlighting is enabled on this query, and when I disable highlighting, the query works rapidly and doesn't kill Elasticsearch. It also only kills Elasticsearch when some particular document or documents are present. I've tried cutting out large swathes of documents, and this makes the query work okay. On the full set, the query reliably kills Elasticsearch. The "and" is causing the query to become more complex, because despite the fact that it's a phrase query, I think the "and" is getting treated as a boolean operator. I tried "health and care", for example, and this sent Elasticsearch into a mini-spiral that it managed to recover from, but returned me results with "health care" as a phrase present, but where each word was highlighted separately. Possibly it's just getting removed as a stop word, but I would think that the highlighting would apply to the phrase as a whole, in that case. So my questions now are: - Is this enough information to hazard any guesses about why Elasticsearch might spiral out of control? - How can I make an exact phrase query on a text field but not have "and" considered a boolean operator? -- Eric On Tue, Jun 14, 2011 at 9:31 PM, Eric Mill [kprojection@gmail.com](mailto:kprojection@gmail.com) wrote: > It's an m1.large, which has high I/O. I'm using the latest version, > elasticsearch v0.16.2. > > I'll try tomorrow morning to see if preventing it from swapping fixes it. > > -- Eric > > On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > Few more questions: > > > > 1. Which ec2 instance type are you using? The smaller you get, the more > > likely they are hurt by "virtualization noise", also, make sure to use ones > > with high IO. > > 2. Which elasticsearch version are you using? > > > > I am not familiar with problems on 10.10, though I heard bad stories about > > it as well (from people using redis) on ec2, but did not encounter it with > > elasticsearch. > > > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > > > 1. I'll try that. > > 2. It's Ubuntu 10.10. Still known problems? > > 3. No, indexing is done in bulk once a night (for now). > > 4. The data directory is on an EBS, but it doesn't appear to be > > misbehaving. I'll investigate a bit more on that front. > > > > Thanks, Shay, > > > > -- Eric > > > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > > > > > wrote: > > > > Some typical things to check: > > > > 1. Make sure the ES (java) process is not swapping. You can force it not > > to swap by setting bootstrap.mlockall to true. > > 2. Which operating system are you running? Ubuntu 10.04 is known to have > > problems on ec2. > > 3. Are you indexing while you do the search? > > 4. Where do you store the data directory? Is it on local drive or EBS? > > Maybe EBS is misbehaving? > > > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > > > Any ideas of why a query might work fine, and then suddenly spike out of > > control? I have the logs, running at DEBUG level, from an example incident. > > The same query gets executed, and on the 10th or so run, the memory starts > > piling up, and top shows the process taking anywhere from 100 to 177% of the > > CPU while this is happening. > > > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > > > I can get it to occur pretty easily after I restart the server, but only > > on my EC2 instance. On my local machine, it's fine. > > > > The query is a dis\_max of 6 text/phrase queries on different fields, on an > > index of about 17,000 documents, where each document ranges from a few > > kilobytes to a few megabytes (not many of those). The data directory takes > > up 2.4GB of space. > > > > Sorry for all the emails - I guess today's my day for questions! > > > > -- Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 16, 2011, 11:11am UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/7 "2011-06-16T11:11:00Z")
- text query does not have boolean operators in it. It simply takes the text, analyzes it, and returns. `and` is a stopword, thats why you don't see it. - how do you do highlighting? Are term vectors stored for the field you highlight on? Is it also stored? - How big are the document and field that you try and highlight on? On Thursday, June 16, 2011 at 2:47 AM, Eric Mill wrote: > OK, I've got a lot more info about the problem. I've now isolated it to a query, not a machine. EC2 isn't the problem, as I first thought it was. I can trigger Elasticsearch to go into a death spiral (where it spikes the CPU and stays there until kill -9'd) the first time this query is run on this set of documents. > > The query is a dis\_max on a set of 6 "text" queries of type "phrase", each for the term "cap and trade". Highlighting is enabled on this query, and when I disable highlighting, the query works rapidly and doesn't kill Elasticsearch. > > It also only kills Elasticsearch when some particular document or documents are present. I've tried cutting out large swathes of documents, and this makes the query work okay. On the full set, the query reliably kills Elasticsearch. > > The "and" is causing the query to become more complex, because despite the fact that it's a phrase query, I think the "and" is getting treated as a boolean operator. I tried "health and care", for example, and this sent Elasticsearch into a mini-spiral that it managed to recover from, but returned me results with "health care" as a phrase present, but where each word was highlighted separately. Possibly it's just getting removed as a stop word, but I would think that the highlighting would apply to the phrase as a whole, in that case. > > So my questions now are: > > - Is this enough information to hazard any guesses about why Elasticsearch might spiral out of control? > > - How can I make an exact phrase query on a text field but not have "and" considered a boolean operator? > > -- Eric > > On Tue, Jun 14, 2011 at 9:31 PM, Eric Mill \<[kprojection@gmail.com](mailto:kprojection@gmail.com) ([mailto:kprojection@gmail.com](mailto:kprojection@gmail.com))\> wrote: > > > It's an m1.large, which has high I/O. I'm using the latest version, elasticsearch v0.16.2. > > > > I'll try tomorrow morning to see if preventing it from swapping fixes it. > > > > -- Eric > > > > On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > > > Few more questions: > > > > > > 1. Which ec2 instance type are you using? The smaller you get, the more likely they are hurt by "virtualization noise", also, make sure to use ones with high IO. > > > 2. Which elasticsearch version are you using? > > > > > > I am not familiar with problems on 10.10, though I heard bad stories about it as well (from people using redis) on ec2, but did not encounter it with elasticsearch. > > > > > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > > > > > > 1. I'll try that. > > > > 2. It's Ubuntu 10.10. Still known problems? > > > > 3. No, indexing is done in bulk once a night (for now). > > > > 4. The data directory is on an EBS, but it doesn't appear to be misbehaving. I'll investigate a bit more on that front. > > > > > > > > Thanks, Shay, > > > > > > > > -- Eric > > > > > > > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > > > > > > > Some typical things to check: > > > > > > > > > > 1. Make sure the ES (java) process is not swapping. You can force it not to swap by setting bootstrap.mlockall to true. > > > > > 2. Which operating system are you running? Ubuntu 10.04 is known to have problems on ec2. > > > > > 3. Are you indexing while you do the search? > > > > > 4. Where do you store the data directory? Is it on local drive or EBS? Maybe EBS is misbehaving? > > > > > > > > > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > > > > > > > > > > Any ideas of why a query might work fine, and then suddenly spike out of control? I have the logs, running at DEBUG level, from an example incident. The same query gets executed, and on the 10th or so run, the memory starts piling up, and top shows the process taking anywhere from 100 to 177% of the CPU while this is happening. > > > > > > > > > > > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > > > > > > > > > > > I can get it to occur pretty easily after I restart the server, but only on my EC2 instance. On my local machine, it's fine. > > > > > > > > > > > > The query is a dis\_max of 6 text/phrase queries on different fields, on an index of about 17,000 documents, where each document ranges from a few kilobytes to a few megabytes (not many of those). The data directory takes up 2.4GB of space. > > > > > > > > > > > > Sorry for all the emails - I guess today's my day for questions! > > > > > > > > > > > > -- Eric ---
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 16, 2011, 4:35pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/8 "2011-06-16T16:35:56Z")
On Thu, Jun 16, 2011 at 7:11 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > - how do you do highlighting? Are term vectors stored for the field you > highlight on? Is it also stored? It's all the defaults; I haven't specified a mapping file. The \_mapping endpoint simply says {type: "string"} for the field in question. Would term\_vector highlighting address the problem? I don't mind a larger index size. Is there any other factor I should keep in mind when deciding whether to use term vectors? I'm new to the world of Lucene and document searching, so it's likely I'm asking naive questions here. > - How big are the document and field that you try and highlight on? In the mapping being searched, there's at least one matching document that is 15MB. The field being searched takes up nearly all that space. There may be other matches, but that's the only humongous one. Thanks again for your help. -- Eric > On Thursday, June 16, 2011 at 2:47 AM, Eric Mill wrote: > > OK, I've got a lot more info about the problem. I've now isolated it to a > query, not a machine. EC2 isn't the problem, as I first thought it was. I > can trigger Elasticsearch to go into a death spiral (where it spikes the CPU > and stays there until kill -9'd) the first time this query is run on this > set of documents. > > The query is a dis\_max on a set of 6 "text" queries of type "phrase", each > for the term "cap and trade". Highlighting is enabled on this query, and > when I disable highlighting, the query works rapidly and doesn't kill > Elasticsearch. > > It also only kills Elasticsearch when some particular document or documents > are present. I've tried cutting out large swathes of documents, and this > makes the query work okay. On the full set, the query reliably kills > Elasticsearch. > > The "and" is causing the query to become more complex, because despite the > fact that it's a phrase query, I think the "and" is getting treated as a > boolean operator. I tried "health and care", for example, and this sent > Elasticsearch into a mini-spiral that it managed to recover from, but > returned me results with "health care" as a phrase present, but where each > word was highlighted separately. Possibly it's just getting removed as a > stop word, but I would think that the highlighting would apply to the phrase > as a whole, in that case. > > So my questions now are: > > - Is this enough information to hazard any guesses about why Elasticsearch > might spiral out of control? > > - How can I make an exact phrase query on a text field but not have "and" > considered a boolean operator? > > -- Eric > > On Tue, Jun 14, 2011 at 9:31 PM, Eric Mill [kprojection@gmail.com](mailto:kprojection@gmail.com) wrote: > > It's an m1.large, which has high I/O. I'm using the latest version, > elasticsearch v0.16.2. > > I'll try tomorrow morning to see if preventing it from swapping fixes it. > > -- Eric > > On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Few more questions: > > 1. Which ec2 instance type are you using? The smaller you get, the more > likely they are hurt by "virtualization noise", also, make sure to use ones > with high IO. > 2. Which elasticsearch version are you using? > > I am not familiar with problems on 10.10, though I heard bad stories about > it as well (from people using redis) on ec2, but did not encounter it with > elasticsearch. > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > 1. I'll try that. > 2. It's Ubuntu 10.10. Still known problems? > 3. No, indexing is done in bulk once a night (for now). > 4. The data directory is on an EBS, but it doesn't appear to be > misbehaving. I'll investigate a bit more on that front. > > Thanks, Shay, > > -- Eric > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Some typical things to check: > > 1. Make sure the ES (java) process is not swapping. You can force it not to > swap by setting bootstrap.mlockall to true. > 2. Which operating system are you running? Ubuntu 10.04 is known to have > problems on ec2. > 3. Are you indexing while you do the search? > 4. Where do you store the data directory? Is it on local drive or EBS? > Maybe EBS is misbehaving? > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > Any ideas of why a query might work fine, and then suddenly spike out of > control? I have the logs, running at DEBUG level, from an example incident. > The same query gets executed, and on the 10th or so run, the memory starts > piling up, and top shows the process taking anywhere from 100 to 177% of the > CPU while this is happening. > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > I can get it to occur pretty easily after I restart the server, but only on > my EC2 instance. On my local machine, it's fine. > > The query is a dis\_max of 6 text/phrase queries on different fields, on an > index of about 17,000 documents, where each document ranges from a few > kilobytes to a few megabytes (not many of those). The data directory takes > up 2.4GB of space. > > Sorry for all the emails - I guess today's my day for questions! > > -- Eric ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 16, 2011, 5:02pm UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/9 "2011-06-16T17:02:19Z")
Yes, if you enable term vector highlighting, then it will be faster to highlight that field. (you will need to reindex the data for that), also, if you store that field, then there won't be a need to load it from \_source and parse it. I suggest creating a simple one node cluster with several of those big documents, and testing it separately. On Thursday, June 16, 2011 at 7:35 PM, Eric Mill wrote: > On Thu, Jun 16, 2011 at 7:11 AM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > - how do you do highlighting? Are term vectors stored for the field you highlight on? Is it also stored? > > It's all the defaults; I haven't specified a mapping file. The \_mapping endpoint simply says {type: "string"} for the field in question. > > Would term\_vector highlighting address the problem? I don't mind a larger index size. Is there any other factor I should keep in mind when deciding whether to use term vectors? > > I'm new to the world of Lucene and document searching, so it's likely I'm asking naive questions here. > > > - How big are the document and field that you try and highlight on? > > In the mapping being searched, there's at least one matching document that is 15MB. The field being searched takes up nearly all that space. There may be other matches, but that's the only humongous one. > > Thanks again for your help. > > -- Eric > > > On Thursday, June 16, 2011 at 2:47 AM, Eric Mill wrote: > > > > > OK, I've got a lot more info about the problem. I've now isolated it to a query, not a machine. EC2 isn't the problem, as I first thought it was. I can trigger Elasticsearch to go into a death spiral (where it spikes the CPU and stays there until kill -9'd) the first time this query is run on this set of documents. > > > > > > The query is a dis\_max on a set of 6 "text" queries of type "phrase", each for the term "cap and trade". Highlighting is enabled on this query, and when I disable highlighting, the query works rapidly and doesn't kill Elasticsearch. > > > > > > It also only kills Elasticsearch when some particular document or documents are present. I've tried cutting out large swathes of documents, and this makes the query work okay. On the full set, the query reliably kills Elasticsearch. > > > > > > The "and" is causing the query to become more complex, because despite the fact that it's a phrase query, I think the "and" is getting treated as a boolean operator. I tried "health and care", for example, and this sent Elasticsearch into a mini-spiral that it managed to recover from, but returned me results with "health care" as a phrase present, but where each word was highlighted separately. Possibly it's just getting removed as a stop word, but I would think that the highlighting would apply to the phrase as a whole, in that case. > > > > > > So my questions now are: > > > > > > - Is this enough information to hazard any guesses about why Elasticsearch might spiral out of control? > > > > > > - How can I make an exact phrase query on a text field but not have "and" considered a boolean operator? > > > > > > -- Eric > > > > > > On Tue, Jun 14, 2011 at 9:31 PM, Eric Mill \<[kprojection@gmail.com](mailto:kprojection@gmail.com) ([mailto:kprojection@gmail.com](mailto:kprojection@gmail.com))\> wrote: > > > > > > > It's an m1.large, which has high I/O. I'm using the latest version, elasticsearch v0.16.2. > > > > > > > > I'll try tomorrow morning to see if preventing it from swapping fixes it. > > > > > > > > -- Eric > > > > > > > > On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > > > > > > > Few more questions: > > > > > > > > > > 1. Which ec2 instance type are you using? The smaller you get, the more likely they are hurt by "virtualization noise", also, make sure to use ones with high IO. > > > > > 2. Which elasticsearch version are you using? > > > > > > > > > > I am not familiar with problems on 10.10, though I heard bad stories about it as well (from people using redis) on ec2, but did not encounter it with elasticsearch. > > > > > > > > > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > > > > > > > > > > 1. I'll try that. > > > > > > 2. It's Ubuntu 10.10. Still known problems? > > > > > > 3. No, indexing is done in bulk once a night (for now). > > > > > > 4. The data directory is on an EBS, but it doesn't appear to be misbehaving. I'll investigate a bit more on that front. > > > > > > > > > > > > Thanks, Shay, > > > > > > > > > > > > -- Eric > > > > > > > > > > > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) ([mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com))\> wrote: > > > > > > > > > > > > > Some typical things to check: > > > > > > > > > > > > > > 1. Make sure the ES (java) process is not swapping. You can force it not to swap by setting bootstrap.mlockall to true. > > > > > > > 2. Which operating system are you running? Ubuntu 10.04 is known to have problems on ec2. > > > > > > > 3. Are you indexing while you do the search? > > > > > > > 4. Where do you store the data directory? Is it on local drive or EBS? Maybe EBS is misbehaving? > > > > > > > > > > > > > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > > > > > > > > > > > > > > Any ideas of why a query might work fine, and then suddenly spike out of control? I have the logs, running at DEBUG level, from an example incident. The same query gets executed, and on the 10th or so run, the memory starts piling up, and top shows the process taking anywhere from 100 to 177% of the CPU while this is happening. > > > > > > > > > > > > > > > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > > > > > > > > > > > > > > > I can get it to occur pretty easily after I restart the server, but only on my EC2 instance. On my local machine, it's fine. > > > > > > > > > > > > > > > > The query is a dis\_max of 6 text/phrase queries on different fields, on an index of about 17,000 documents, where each document ranges from a few kilobytes to a few megabytes (not many of those). The data directory takes up 2.4GB of space. > > > > > > > > > > > > > > > > Sorry for all the emails - I guess today's my day for questions! > > > > > > > > > > > > > > > > -- Eric ---
**Author:** ![Eric\_Mill](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_mill/32/3124_2.png) [@Eric\_Mill](https://discuss.elastic.co/u/Eric_Mill)\ **Post date:** [June 18, 2011, 5:09am UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/10 "2011-06-18T05:09:25Z")
So I did both of these (stored the field on the index, and enabled term vector highlighting (and reindexed)), and for most queries, this speeds up highlighting, and puts the server under less load. The query that used to kill Elasticsearch ("cap and trade") now is quite fast. There are some queries that were fine before, that now exhibit higher strain. I've (only) once, and not reproducibly, sent ES into a death spiral since with one of them (a one word query ("health"). This is probably fine, for now, but is going to be a problem in the long run. I know I haven't given you a lot of detail about my current setup, but can you think of any other general optimizations I could perform besides enabling term vector highlighting and storing the fields being searched on the index? -- Eric On Thu, Jun 16, 2011 at 1:02 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Yes, if you enable term vector highlighting, then it will be faster to > highlight that field. (you will need to reindex the data for that), also, if > you store that field, then there won't be a need to load it from \_source and > parse it. I suggest creating a simple one node cluster with several of those > big documents, and testing it separately. > > On Thursday, June 16, 2011 at 7:35 PM, Eric Mill wrote: > > On Thu, Jun 16, 2011 at 7:11 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > - how do you do highlighting? Are term vectors stored for the field you > highlight on? Is it also stored? > > It's all the defaults; I haven't specified a mapping file. The \_mapping > endpoint simply says {type: "string"} for the field in question. > > Would term\_vector highlighting address the problem? I don't mind a larger > index size. Is there any other factor I should keep in mind when deciding > whether to use term vectors? > > I'm new to the world of Lucene and document searching, so it's likely I'm > asking naive questions here. > > - How big are the document and field that you try and highlight on? > > In the mapping being searched, there's at least one matching document that > is 15MB. The field being searched takes up nearly all that space. There may > be other matches, but that's the only humongous one. > > Thanks again for your help. > > -- Eric > > On Thursday, June 16, 2011 at 2:47 AM, Eric Mill wrote: > > OK, I've got a lot more info about the problem. I've now isolated it to a > query, not a machine. EC2 isn't the problem, as I first thought it was. I > can trigger Elasticsearch to go into a death spiral (where it spikes the CPU > and stays there until kill -9'd) the first time this query is run on this > set of documents. > > The query is a dis\_max on a set of 6 "text" queries of type "phrase", each > for the term "cap and trade". Highlighting is enabled on this query, and > when I disable highlighting, the query works rapidly and doesn't kill > Elasticsearch. > > It also only kills Elasticsearch when some particular document or documents > are present. I've tried cutting out large swathes of documents, and this > makes the query work okay. On the full set, the query reliably kills > Elasticsearch. > > The "and" is causing the query to become more complex, because despite the > fact that it's a phrase query, I think the "and" is getting treated as a > boolean operator. I tried "health and care", for example, and this sent > Elasticsearch into a mini-spiral that it managed to recover from, but > returned me results with "health care" as a phrase present, but where each > word was highlighted separately. Possibly it's just getting removed as a > stop word, but I would think that the highlighting would apply to the phrase > as a whole, in that case. > > So my questions now are: > > - Is this enough information to hazard any guesses about why Elasticsearch > might spiral out of control? > > - How can I make an exact phrase query on a text field but not have "and" > considered a boolean operator? > > -- Eric > > On Tue, Jun 14, 2011 at 9:31 PM, Eric Mill [kprojection@gmail.com](mailto:kprojection@gmail.com) wrote: > > It's an m1.large, which has high I/O. I'm using the latest version, > elasticsearch v0.16.2. > > I'll try tomorrow morning to see if preventing it from swapping fixes it. > > -- Eric > > On Tue, Jun 14, 2011 at 7:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Few more questions: > > 1. Which ec2 instance type are you using? The smaller you get, the more > likely they are hurt by "virtualization noise", also, make sure to use ones > with high IO. > 2. Which elasticsearch version are you using? > > I am not familiar with problems on 10.10, though I heard bad stories about > it as well (from people using redis) on ec2, but did not encounter it with > elasticsearch. > > On Wednesday, June 15, 2011 at 1:59 AM, Eric Mill wrote: > > 1. I'll try that. > 2. It's Ubuntu 10.10. Still known problems? > 3. No, indexing is done in bulk once a night (for now). > 4. The data directory is on an EBS, but it doesn't appear to be > misbehaving. I'll investigate a bit more on that front. > > Thanks, Shay, > > -- Eric > > On Tue, Jun 14, 2011 at 4:25 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > Some typical things to check: > > 1. Make sure the ES (java) process is not swapping. You can force it not to > swap by setting bootstrap.mlockall to true. > 2. Which operating system are you running? Ubuntu 10.04 is known to have > problems on ec2. > 3. Are you indexing while you do the search? > 4. Where do you store the data directory? Is it on local drive or EBS? > Maybe EBS is misbehaving? > > On Tuesday, June 14, 2011 at 10:28 PM, Eric Mill wrote: > > Any ideas of why a query might work fine, and then suddenly spike out of > control? I have the logs, running at DEBUG level, from an example incident. > The same query gets executed, and on the 10th or so run, the memory starts > piling up, and top shows the process taking anywhere from 100 to 177% of the > CPU while this is happening. > > [ElasticSearch running out of control · GitHub](https://gist.github.com/1025642) > > I can get it to occur pretty easily after I restart the server, but only on > my EC2 instance. On my local machine, it's fine. > > The query is a dis\_max of 6 text/phrase queries on different fields, on an > index of about 17,000 documents, where each document ranges from a few > kilobytes to a few megabytes (not many of those). The data directory takes > up 2.4GB of space. > > Sorry for all the emails - I guess today's my day for questions! > > -- Eric ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:03am UTC](https://discuss.elastic.co/t/cpu-and-memory-usage-suddenly-spiral-out-of-control/4621/11 "2017-07-06T04:03:13Z")