Java.lang.OutOfMemoryError: Java heap space

Elastic Stack Elasticsearch
1

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option.
The elasticsearch.yml is not in the classpath which means it will go for
defaults. I created 1 index seeded 3 docs and the size of each doc is not
more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][T#1]"
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

The node I created has these settings

NodeBuilder nb = new NodeBuilder().clusterName("my-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);
nb.settings().put("ES_MIN_MEM", "2g");
nb.settings().put("ES_MAX_MEM", "2g");

Node node = nb.client(false).node();

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option.
The elasticsearch.yml is not in the classpath which means it will go for
defaults. I created 1 index seeded 3 docs and the size of each doc is not
more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][T#1]"
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

It won't work.
Your JVM heap size must be set outside your program.

Typically, set JAVA_OPTS=-Xmx2g -Xms2g or something like that.
Then run your program. The JVM will allocate memory for it.
Embedded ES nodes will use it.

Make sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 juin 2013 à 19:54, vinod eligeti veligeti999@gmail.com a écrit :

The node I created has these settings

			NodeBuilder nb = new NodeBuilder().clusterName("my-store");
			nb.settings().put("node.data", true);
			nb.settings().put("index.number_of_shards", 1);
			nb.settings().put("index.number_of_replicas", 0);
			nb.settings().put("ES_MIN_MEM", "2g");
			nb.settings().put("ES_MAX_MEM", "2g");

			Node node = nb.client(false).node();

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option. The elasticsearch.yml is not in the classpath which means it will go for defaults. I created 1 index seeded 3 docs and the size of each doc is not more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][T#1]" java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

And why are you requesting size 2147483646???

On 18 June 2013 21:24, David Pilato david@pilato.fr wrote:

It won't work.
Your JVM heap size must be set outside your program.

Typically, set JAVA_OPTS=-Xmx2g -Xms2g or something like that.
Then run your program. The JVM will allocate memory for it.
Embedded ES nodes will use it.

Make sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 juin 2013 à 19:54, vinod eligeti veligeti999@gmail.com a écrit :

The node I created has these settings

NodeBuilder nb = new NodeBuilder().clusterName("my-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);
nb.settings().put("ES_MIN_MEM", "2g");
nb.settings().put("ES_MAX_MEM", "2g");

Node node = nb.client(false).node();

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option.
The elasticsearch.yml is not in the classpath which means it will go for
defaults. I created 1 index seeded 3 docs and the size of each doc is not
more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][**T#1]"
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.search.**SearchService.shortcutDocIdsToLoad(
SearchService.java:588)
at org.elasticsearch.search.**SearchService.executeFetchPhase(
SearchService.java:325)
at org.elasticsearch.search.**action.SearchServiceTransportAction.
sendExecuteFetch(**SearchServiceTransportAction.java:243)
at org.elasticsearch.action.search.type.
TransportSearchQueryAndFetchAction$AsyncAction.**sendExecuteFirstPhase(
TransportSearchQueryAndFetchAction.java:75)
at org.elasticsearch.action.**search.type.TransportSearchTypeAction$
BaseAsyncAction.**performFirstPhase(TransportSearchTypeAction.
java:205)
at org.elasticsearch.action.**search.type.TransportSearchTypeAction$
BaseAsyncAction.**performFirstPhase(TransportSearchTypeAction.
java:192)
at org.elasticsearch.action.**search.type.TransportSearchTypeAction$
BaseAsyncAction$2.run(**TransportSearchTypeAction.**java:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Yes and I figured that is the problem. Sorry not to update soon.

On Wed, Jun 19, 2013 at 12:40 PM, Clinton Gormley clint@traveljury.comwrote:

And why are you requesting size 2147483646???

On 18 June 2013 21:24, David Pilato david@pilato.fr wrote:

It won't work.
Your JVM heap size must be set outside your program.

Typically, set JAVA_OPTS=-Xmx2g -Xms2g or something like that.
Then run your program. The JVM will allocate memory for it.
Embedded ES nodes will use it.

Make sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 juin 2013 à 19:54, vinod eligeti veligeti999@gmail.com a écrit :

The node I created has these settings

NodeBuilder nb = new NodeBuilder().clusterName("my-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);
nb.settings().put("ES_MIN_MEM", "2g");
nb.settings().put("ES_MAX_MEM", "2g");

Node node = nb.client(false).node();

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option.
The elasticsearch.yml is not in the classpath which means it will go for
defaults. I created 1 index seeded 3 docs and the size of each doc is not
more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][**T#1]"
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.search.**SearchService.shortcutDocIdsToLoad(
SearchService.java:588)
at org.elasticsearch.search.**SearchService.executeFetchPhase(
SearchService.java:325)
at org.elasticsearch.search.**action.SearchServiceTransportAction.
sendExecuteFetch(SearchServiceTransportAction.java:243)
at org.elasticsearch.action.search.type.
TransportSearchQueryAndFetchAction$AsyncAction.
sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at org.elasticsearch.action.**search.type.*TransportSearchTypeAction$
*BaseAsyncAction.**performFirstPhase(TransportSearchTypeAction.
java:205)
at org.elasticsearch.action.**search.type.*TransportSearchTypeAction$
*BaseAsyncAction.**performFirstPhase(TransportSearchTypeAction.
java:192)
at org.elasticsearch.action.**search.type.*TransportSearchTypeAction$
*BaseAsyncAction$2.run(**TransportSearchTypeAction.**java:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.
runTask(ThreadPoolExecutor.**java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.**java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Oh my god! I completely missed that!!!

Thanks Clint!

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 19 juin 2013 à 21:40, Clinton Gormley clint@traveljury.com a écrit :

And why are you requesting size 2147483646???

On 18 June 2013 21:24, David Pilato david@pilato.fr wrote:

It won't work.
Your JVM heap size must be set outside your program.

Typically, set JAVA_OPTS=-Xmx2g -Xms2g or something like that.
Then run your program. The JVM will allocate memory for it.
Embedded ES nodes will use it.

Make sense?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 juin 2013 à 19:54, vinod eligeti veligeti999@gmail.com a écrit :

The node I created has these settings

  		NodeBuilder nb = new NodeBuilder().clusterName("my-store");
  		nb.settings().put("node.data", true);
  		nb.settings().put("index.number_of_shards", 1);
  		nb.settings().put("index.number_of_replicas", 0);
  		nb.settings().put("ES_MIN_MEM", "2g");
  		nb.settings().put("ES_MAX_MEM", "2g");

  		Node node = nb.client(false).node();

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option. The elasticsearch.yml is not in the classpath which means it will go for defaults. I created 1 index seeded 3 docs and the size of each doc is not more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][T#1]" java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

When you request an extremely large size query result (in your case,
apparently Integer.MAX_VALUE -1)
Elasticsearch will attempt to allocate an array of that size. You might
have a look at the source line where the OOM occurs, and you will see it,
an allocation of int[context.size()], the equivalent of new
int[Integer.MAX_VALUE -1].
So the idiom of setting a huge query size in order to "select everything"
won't work for Elasticsearch. You either need to know the max size of your
query (perhaps doing a count query first) or use the scan/scroll API.

On Tuesday, June 18, 2013 10:52:12 AM UTC-7, vinod eligeti wrote:

I am getting heap space error.

Basically I am starting a embedded ES node with data set to true option.
The elasticsearch.yml is not in the classpath which means it will go for
defaults. I created 1 index seeded 3 docs and the size of each doc is not
more than 1000 bytes. And when I ran this query

query: {
"size" : 2147483646,
"query" : {
"bool" : {
"should" : {
"term" : {
"account-name" : "customer1"
}
},
"minimum_should_match" : "1"
}
},
"filter" : {
"range" : {
"timestamp" : {
"from" : 1371577346230,
"to" : 1371577406229,
"include_lower" : true,
"include_upper" : true
}
}
}
}

And then i get this:

Exception in thread "elasticsearch[Hulk][search][T#1]"
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.search.SearchService.shortcutDocIdsToLoad(SearchService.java:588)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:325)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:243)
at
org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:75)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:205)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:192)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2.run(TransportSearchTypeAction.java:178)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

Why such a small data I am getting Java heap error.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like
8

Hi,
I have the same problem. I got a single node which has a tomcat frontend
where I just created a servlet to convert a specific frontend query
language to ES JSON queries.
Everything works quite fine, until I use JMeter to test performance of the
system. At that point I get, after several queries, an OutOfMemoryError.
I tried increasing memory heap up to 2g but nothing. The index on disk,
actually, takes less than a GB, so what's happening?
My queries deal with filters and query strings, no facets at the moment.

I double checked that all the suggestions seen in this post were applied
(size of result = 30, 2g of heap). At the moment as I increase the memory
for higher values I get an aborted, due to memory limits per process but,
anyway, the index is 900Mb! Why would it occupy in memory more than this?
Just for testing, is there any kind of query caching that may be disabled?
The aim of the project is to index hundreds of GBs of emails... Actually we
got not more than a 1000 mails for the indicated index size...

Can you help me?
Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

If you use JMeter, that means that you are testing your PROD platform?
And you won't give more than 2Gb heap?

How many concurrent requests did you send?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2013 à 07:03, Massimiliano Perantoni massimiliano.perantoni@gmail.com a écrit :

Hi,
I have the same problem. I got a single node which has a tomcat frontend where I just created a servlet to convert a specific frontend query language to ES JSON queries.
Everything works quite fine, until I use JMeter to test performance of the system. At that point I get, after several queries, an OutOfMemoryError.
I tried increasing memory heap up to 2g but nothing. The index on disk, actually, takes less than a GB, so what's happening?
My queries deal with filters and query strings, no facets at the moment.

I double checked that all the suggestions seen in this post were applied (size of result = 30, 2g of heap). At the moment as I increase the memory for higher values I get an aborted, due to memory limits per process but, anyway, the index is 900Mb! Why would it occupy in memory more than this? Just for testing, is there any kind of query caching that may be disabled?
The aim of the project is to index hundreds of GBs of emails... Actually we got not more than a 1000 mails for the indicated index size...

Can you help me?
Thanks!

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Well, I'd give it even more, but as soon as I reach 4G of heap the
process get aborted...
That's actually a POC platform, but we're sizing the PROD system on
it. I have to understand if it's a "hardware" limit or if it's a
leak... Up til now I noticed that as I grow it takes.
I don't actually know how ES works in allocating memory, but I'd
expect that memorìy usage it should not overtake the index size... Is
it that wrong as an assuption?

java -version
java version "1.7.0_21"
uname -a
Linux mercury 3.10.10-200.fc19.x86_64 #1 SMP Thu Aug 29 19:05:45 UTC
2013 x86_64 x86_64 x86_64 GNU/Linux

ulimit -u
unlimited
ulimit -l
unlimited
everything, to do a first test, is running as root.
Thanks for any help!

2013/10/3 David Pilato david@pilato.fr:

If you use JMeter, that means that you are testing your PROD platform?
And you won't give more than 2Gb heap?

How many concurrent requests did you send?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2013 à 07:03, Massimiliano Perantoni
massimiliano.perantoni@gmail.com a écrit :

Hi,
I have the same problem. I got a single node which has a tomcat frontend
where I just created a servlet to convert a specific frontend query language
to ES JSON queries.
Everything works quite fine, until I use JMeter to test performance of the
system. At that point I get, after several queries, an OutOfMemoryError.
I tried increasing memory heap up to 2g but nothing. The index on disk,
actually, takes less than a GB, so what's happening?
My queries deal with filters and query strings, no facets at the moment.

I double checked that all the suggestions seen in this post were applied
(size of result = 30, 2g of heap). At the moment as I increase the memory
for higher values I get an aborted, due to memory limits per process but,
anyway, the index is 900Mb! Why would it occupy in memory more than this?
Just for testing, is there any kind of query caching that may be disabled?
The aim of the project is to index hundreds of GBs of emails... Actually we
got not more than a 1000 mails for the indicated index size...

Can you help me?
Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Just to be complete, the concurrency just delays or anticipates the Error...
If I use just 1 connection or 20 connections I get the "system death"
earlier or later.
As the web and the ES engine live on the same system, actually I had
to reduce the number of connections to ES, as concurrency brought the
system to socket starvation.
One more thing... I'm using this way to connect...
Client client = new TransportClient()
.addTransportAddress(new InetSocketTransportAddress(
"127.0.0.1", 9300));
[.... doing my query...]
client.close();

everything in the servlet. As soon as I close the client, I'd expect
that when the method using it completes the GC cleans it (some day,
not immediately), but that the close() method closes the socket...
Always using JMeter we noticed that this is not true... We get a
socket starvation after some minutes of testing. After that I started
using a connectionpool, with 20 connections, and everything is working
OK, but now we get the OOME....
Thanks!

2013/10/3 David Pilato david@pilato.fr:

If you use JMeter, that means that you are testing your PROD platform?
And you won't give more than 2Gb heap?

How many concurrent requests did you send?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 oct. 2013 à 07:03, Massimiliano Perantoni
massimiliano.perantoni@gmail.com a écrit :

Hi,
I have the same problem. I got a single node which has a tomcat frontend
where I just created a servlet to convert a specific frontend query language
to ES JSON queries.
Everything works quite fine, until I use JMeter to test performance of the
system. At that point I get, after several queries, an OutOfMemoryError.
I tried increasing memory heap up to 2g but nothing. The index on disk,
actually, takes less than a GB, so what's happening?
My queries deal with filters and query strings, no facets at the moment.

I double checked that all the suggestions seen in this post were applied
(size of result = 30, 2g of heap). At the moment as I increase the memory
for higher values I get an aborted, due to memory limits per process but,
anyway, the index is 900Mb! Why would it occupy in memory more than this?
Just for testing, is there any kind of query caching that may be disabled?
The aim of the project is to index hundreds of GBs of emails... Actually we
got not more than a 1000 mails for the indicated index size...

Can you help me?
Thanks!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

You assumption the index file size on disk should match maximum heap is
wrong.

First, the index is bigger than on disk, since compression is used. It may
be a factor of 5 or 10 or even more.

Second, more important, the index never loads on the heap, but into the
filesystem cache.

If you observe OOM there is something wrong with the query.

Can you please show the queries you use, and the full stack trace?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

In addition to this, make sure that you have only one TransportClient for the full JVM. You don't need and you should not create many clients.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 oct. 2013 à 11:58, "joergprante@gmail.com" joergprante@gmail.com a écrit :

You assumption the index file size on disk should match maximum heap is wrong.

First, the index is bigger than on disk, since compression is used. It may be a factor of 5 or 10 or even more.

Second, more important, the index never loads on the heap, but into the filesystem cache.

If you observe OOM there is something wrong with the query.

Can you please show the queries you use, and the full stack trace?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

Hi,
dealing with Client number the problem doesn't change that much. If I
just connect once (I create a pool with just one connection), I'll get
the same problem, after some time the system is running.
Just to be sure that I understood... Should I call the close() method
each time I end a query? Or I should close it just when I close the
client application?
dealing with queries, these are various queries that I do, that would
be a big log... Can I send it? Which other information may be useful
to you?
Ciao!

2013/10/3 David Pilato david@pilato.fr:

In addition to this, make sure that you have only one TransportClient for
the full JVM. You don't need and you should not create many clients.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 oct. 2013 à 11:58, "joergprante@gmail.com" joergprante@gmail.com a
écrit :

You assumption the index file size on disk should match maximum heap is
wrong.

First, the index is bigger than on disk, since compression is used. It may
be a factor of 5 or 10 or even more.

Second, more important, the index never loads on the heap, but into the
filesystem cache.

If you observe OOM there is something wrong with the query.

Can you please show the queries you use, and the full stack trace?

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

Only call close when you close the client app.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 oct. 2013 à 16:24, Massimiliano Perantoni massimiliano.perantoni@gmail.com a écrit :

Hi,
dealing with Client number the problem doesn't change that much. If I
just connect once (I create a pool with just one connection), I'll get
the same problem, after some time the system is running.
Just to be sure that I understood... Should I call the close() method
each time I end a query? Or I should close it just when I close the
client application?
dealing with queries, these are various queries that I do, that would
be a big log... Can I send it? Which other information may be useful
to you?
Ciao!

2013/10/3 David Pilato david@pilato.fr:

In addition to this, make sure that you have only one TransportClient for
the full JVM. You don't need and you should not create many clients.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 3 oct. 2013 à 11:58, "joergprante@gmail.com" joergprante@gmail.com a
écrit :

You assumption the index file size on disk should match maximum heap is
wrong.

First, the index is bigger than on disk, since compression is used. It may
be a factor of 5 or 10 or even more.

Second, more important, the index never loads on the heap, but into the
filesystem cache.

If you observe OOM there is something wrong with the query.

Can you please show the queries you use, and the full stack trace?

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

16

For what it's worth, i am searching 1m documents, about 5G, using a max
heap of 350m and its all working just fine. This includes facets over the
entire index...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

17

Have you ever stressed the engine to test its reactions?

2013/10/3 James Richardson james.time4tea@gmail.com:

For what it's worth, i am searching 1m documents, about 5G, using a max heap
of 350m and its all working just fine. This includes facets over the entire
index...

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

18

Well, I have tested that it works with the load I need. I don't really see
the point in creating some imaginary load levels and just stressing it out.
If it works for what I needs that's good enough... The more important thing
(for me in this instance) is that it works enough and the heap size is
small enough that it can run on a cheap vm.

James

On 3 October 2013 16:47, Massimiliano Perantoni <
massimiliano.perantoni@gmail.com> wrote:

Have you ever stressed the engine to test its reactions?

2013/10/3 James Richardson james.time4tea@gmail.com:

For what it's worth, i am searching 1m documents, about 5G, using a max
heap
of 350m and its all working just fine. This includes facets over the
entire
index...

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

19

Actually we'll have to stand the load of hundreds of simultaneous queries...

2013/10/3 James Richardson james@time4tea.net:

Well, I have tested that it works with the load I need. I don't really see
the point in creating some imaginary load levels and just stressing it out.
If it works for what I needs that's good enough... The more important thing
(for me in this instance) is that it works enough and the heap size is
small enough that it can run on a cheap vm.

James

On 3 October 2013 16:47, Massimiliano Perantoni
massimiliano.perantoni@gmail.com wrote:

Have you ever stressed the engine to test its reactions?

2013/10/3 James Richardson james.time4tea@gmail.com:

For what it's worth, i am searching 1m documents, about 5G, using a max
heap
of 350m and its all working just fine. This includes facets over the
entire
index...

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

20

I quite often find that the real number of actually simultaneous things is
often fewer than you might first think - each query only lasts a few ms,
often the results can be cached using squid or varnish, lots of times
people are searching common content for the same stuff, each of which
reduces the number of simultaneous queries that you have to do. In order to
support 100's of users, you might find that really only 1 or 2 queries are
actually happening at the same time... I mean YMMV...

On 3 October 2013 16:56, Massimiliano Perantoni <
massimiliano.perantoni@gmail.com> wrote:

Actually we'll have to stand the load of hundreds of simultaneous
queries...

2013/10/3 James Richardson james@time4tea.net:

Well, I have tested that it works with the load I need. I don't really
see
the point in creating some imaginary load levels and just stressing it
out.
If it works for what I needs that's good enough... The more important
thing
(for me in this instance) is that it works enough and the heap size is
small enough that it can run on a cheap vm.

James

On 3 October 2013 16:47, Massimiliano Perantoni
massimiliano.perantoni@gmail.com wrote:

Have you ever stressed the engine to test its reactions?

2013/10/3 James Richardson james.time4tea@gmail.com:

For what it's worth, i am searching 1m documents, about 5G, using a
max
heap
of 350m and its all working just fine. This includes facets over the
entire
index...

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit

https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.

To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/CfQsLQWqGRg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.