CompileException - out of heap space - tried to access class ScriptDocValues$Strings$1 from class ASMAccessorImpl

Elastic Stack Elasticsearch
1

I'm trying to boost my query to prefer documents with a longer array for a
certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.matchQuery("myField1", searchFieldValue1));
query.should(QueryBuilders.matchQuery("myField2", searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].values.length");

// Create final scoring function
FunctionScoreQueryBuilder function =

QueryBuilders.functionScoreQuery(query, script);

This gives me the following errors:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][3]:
QueryPhaseExecutionException[[myIndex][3]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][4]: ElasticSearchException[tried to
access class org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1
from class ASMAccessorImpl_4033620681383178478830]; nested:
IllegalAccessError[tried to access class
org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1 from class
ASMAccessorImpl_4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][1]:
QueryPhaseExecutionException[[myIndex][1]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][2]:
QueryPhaseExecutionException[[myIndex][2]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][0]: ElasticSearchException[tried to
access class org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1
from class ASMAccessorImpl_3774454791383178471610]; nested:
IllegalAccessError[tried to access class
org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1 from class
ASMAccessorImpl_3774454791383178471610]; }

Thanks,
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array for a
certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.matchQuery("myField1", searchFieldValue1));
query.should(QueryBuilders.matchQuery("myField2", searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].values.length");

// Create final scoring function
FunctionScoreQueryBuilder function =

QueryBuilders.functionScoreQuery(query, script);

This gives me the following errors:

org.elasticsearch.action.search.SearchPhaseExecutionException: Failed to
execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][3]:
QueryPhaseExecutionException[[myIndex][3]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][4]: ElasticSearchException[tried to
access class org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1
from class ASMAccessorImpl_4033620681383178478830]; nested:
IllegalAccessError[tried to access class
org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1 from class
ASMAccessorImpl_4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][1]:
QueryPhaseExecutionException[[myIndex][1]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][2]:
QueryPhaseExecutionException[[myIndex][2]: query[filtered(function score
(myField1:searchFieldValue1,function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed to
execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested:
ElasticSearchException[java.lang.OutOfMemoryError: Java heap space];
nested: ExecutionError[java.lang.OutOfMemoryError: Java heap space];
nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][myIndex][0]: ElasticSearchException[tried to
access class org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1
from class ASMAccessorImpl_3774454791383178471610]; nested:
IllegalAccessError[tried to access class
org.elasticsearch.index.fielddata.ScriptDocValues$Strings$1 from class
ASMAccessorImpl_3774454791383178471610]; }

Thanks,
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey Ben,

you ran out of heap/memory. The simplest solution is always to add more
memory to your nodes - ok, simplest solution from an application point of
view :slight_smile:

Obviously it would be great, if we can find out, why the OOM exception
actually happened. Question is, did the exception happen, because you
executed/compiled a script or did it already happen before. You should
check your logs for that.

If it happened before, you should also find out why (facetting on an
analyzed field, sorting on data which does not fit in memory etc). So you
will have to query nodes stats/infos APIs to find more about your running
system.

Another thing is, that you might want to try with an oracle JDK and see if
you get the same problems.

--Alex

On Thu, Oct 31, 2013 at 2:39 AM, Ben McCann benjamin.j.mccann@gmail.comwrote:

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though
IllegalAccessError when using an MVEL script filter in the query · Issue #3094 · elastic/elasticsearch · GitHub

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10) (7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array for
a certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.**matchQuery("myField1",

searchFieldValue1));
query.should(QueryBuilders.**matchQuery("myField2",
searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].**values.length");

// Create final scoring function
FunctionScoreQueryBuilder function = QueryBuilders.**functionScoreQuery(query,

script);

This gives me the following errors:

org.elasticsearch.action.**search.**SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][3]: QueryPhaseExecutionException[[**myIndex][3]:
query[filtered(function score (myField1:searchFieldValue1,**function=script[_score

  • doc['myArray.myObjField'].values.length], params
    [null]))->cache(_type:person)]    ,from[0],size[10]: Query Failed [Failed
    to execute main query]]; nested: CompileException[[Error:
    java.lang.OutOfMemoryError: Java heap space]
    [Near : {... score * doc['myArray.my ....}]
    ^
    [Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
    Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
    Java heap space]; nested: OutOfMemoryError[Java heap space];
    }{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][4]: ElasticSearchException[tried
    to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
    from class ASMAccessorImpl    4033620681383178478830]; nested:
    IllegalAccessError[tried to access class org.elasticsearch.index.
    fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl_**4033620681383178478830];
    }{[Bfi-HZIRQuWIZ6_ApL7sag][*myIndex][1]: QueryPhaseExecutionException[[
    myIndex][1]: query[filtered(function score (myField1:searchFieldValue1,
    *function=script[_score * doc['myArray.myObjField'].values.length],
    params [null]))->cache(_type:person)]    ,from[0],size[10]: Query Failed
    [Failed to execute main query]]; nested: CompileException[[Error:
    java.lang.OutOfMemoryError: Java heap space]
    [Near : {... _score * doc['myArray.my ....}]
    ^
    [Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
    Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
    Java heap space]; nested: OutOfMemoryError[Java heap space];
    }{[Bfi-HZIRQuWIZ6_ApL7sag][*myIndex][2]: QueryPhaseExecutionException[[
    myIndex][2]: query[filtered(function score (myField1:searchFieldValue1,
    *function=script[score * doc['myArray.myObjField'].values.length],
    params [null]))->cache(_type:person)]    ,from[0],size[10]: Query Failed
    [Failed to execute main query]]; nested: CompileException[[Error:
    java.lang.OutOfMemoryError: Java heap space]
    [Near : {... score * doc['myArray.my ....}]
    ^
    [Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
    Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
    Java heap space]; nested: OutOfMemoryError[Java heap space];
    }{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][0]: ElasticSearchException[tried
    to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
    from class ASMAccessorImpl    3774454791383178471610]; nested:
    IllegalAccessError[tried to access class org.elasticsearch.index.
    fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl    **3774454791383178471610];
    }

Thanks,
Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi Alex,

I appreciate the response. It sounds like you're suggesting that if I get
an OOM exception once, than all future requests will fail stating there's
been an OOM exception? That's not the case for me. I can run other queries
and only this one fails, so it seems pretty clear the problem is with this
query specifically.

I've attached the log from my elasticsearch server, which has some more
information about the problem. I'm really stuck on this, so I'm hoping your
expertise will get me pointed in the right direction :slight_smile:

Thanks,
Ben

On Thu, Oct 31, 2013 at 3:44 AM, Alexander Reelsen alr@spinscale.de wrote:

Hey Ben,

you ran out of heap/memory. The simplest solution is always to add more
memory to your nodes - ok, simplest solution from an application point of
view :slight_smile:

Obviously it would be great, if we can find out, why the OOM exception
actually happened. Question is, did the exception happen, because you
executed/compiled a script or did it already happen before. You should
check your logs for that.

If it happened before, you should also find out why (facetting on an
analyzed field, sorting on data which does not fit in memory etc). So you
will have to query nodes stats/infos APIs to find more about your running
system.

Another thing is, that you might want to try with an oracle JDK and see if
you get the same problems.

--Alex

On Thu, Oct 31, 2013 at 2:39 AM, Ben McCann benjamin.j.mccann@gmail.comwrote:

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though
IllegalAccessError when using an MVEL script filter in the query · Issue #3094 · elastic/elasticsearch · GitHub

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10)
(7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array for
a certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.**matchQuery("myField1",

searchFieldValue1));
query.should(QueryBuilders.**matchQuery("myField2",
searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].**values.length");

// Create final scoring function
FunctionScoreQueryBuilder function = QueryBuilders.**functionScoreQuery(query,

script);

This gives me the following errors:

org.elasticsearch.action.**search.**SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][*myIndex][3]: QueryPhaseExecutionException[[
*myIndex][3]: query[filtered(function score (myField1:searchFieldValue1,
**function=script[score * doc['myArray.myObjField'].values.length],
params [null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][4]: ElasticSearchException[tried
to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
from class ASMAccessorImpl4033620681383178478830]; nested:
IllegalAccessError[tried to access class org.elasticsearch.index.
fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][1]: QueryPhaseExecutionException[[
**myIndex][1]: query[filtered(function score
(myField1:searchFieldValue1,**function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed
to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][2]: QueryPhaseExecutionException[[
**myIndex][2]: query[filtered(function score
(myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed [Failed
to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][0]: ElasticSearchException[tried
to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
from class ASMAccessorImpl3774454791383178471610]; nested:
IllegalAccessError[tried to access class org.elasticsearch.index.
fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
}

Thanks,
Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I just installed Oracle JDK and am having the same problem. Let me know if
there's anything else I can do to help debug this.

Thanks,
Ben

On Thu, Oct 31, 2013 at 12:16 PM, Ben McCann ben@benmccann.com wrote:

Hi Alex,

I appreciate the response. It sounds like you're suggesting that if I get
an OOM exception once, than all future requests will fail stating there's
been an OOM exception? That's not the case for me. I can run other queries
and only this one fails, so it seems pretty clear the problem is with this
query specifically.

I've attached the log from my elasticsearch server, which has some more
information about the problem. I'm really stuck on this, so I'm hoping your
expertise will get me pointed in the right direction :slight_smile:

Thanks,
Ben

On Thu, Oct 31, 2013 at 3:44 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey Ben,

you ran out of heap/memory. The simplest solution is always to add more
memory to your nodes - ok, simplest solution from an application point of
view :slight_smile:

Obviously it would be great, if we can find out, why the OOM exception
actually happened. Question is, did the exception happen, because you
executed/compiled a script or did it already happen before. You should
check your logs for that.

If it happened before, you should also find out why (facetting on an
analyzed field, sorting on data which does not fit in memory etc). So you
will have to query nodes stats/infos APIs to find more about your running
system.

Another thing is, that you might want to try with an oracle JDK and see
if you get the same problems.

--Alex

On Thu, Oct 31, 2013 at 2:39 AM, Ben McCann benjamin.j.mccann@gmail.comwrote:

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though
IllegalAccessError when using an MVEL script filter in the query · Issue #3094 · elastic/elasticsearch · GitHub

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10)
(7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array
for a certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.**matchQuery("myField1",

searchFieldValue1));
query.should(QueryBuilders.**matchQuery("myField2",
searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new

ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].**
values.length");

// Create final scoring function
FunctionScoreQueryBuilder function = QueryBuilders.**functionScoreQuery(query,

script);

This gives me the following errors:

org.elasticsearch.action.**search.**SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][3]: QueryPhaseExecutionException[[
**myIndex][3]: query[filtered(function score
(myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][4]: ElasticSearchException[tried
to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
from class ASMAccessorImpl4033620681383178478830]; nested:
IllegalAccessError[tried to access class org.elasticsearch.index.
fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][1]:
QueryPhaseExecutionException[[**myIndex][1]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][2]:
QueryPhaseExecutionException[[**myIndex][2]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][0]: ElasticSearchException[tried
to access class org.elasticsearch.index.**fielddata.ScriptDocValues$**Strings$1
from class ASMAccessorImpl3774454791383178471610]; nested:
IllegalAccessError[tried to access class org.elasticsearch.index.
fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
}

Thanks,
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Ben McCann - Mountain View, California, Connectifier, Carnegie Mellon University | about.me

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Hey,

only quick checking your log, it seems the exception stems from

at
org.elasticsearch.index.fielddata.AbstractIndexFieldData.load(AbstractIndexFieldData.java:60)

which means, it did occur while you tried to load data into memory which
was too big.

Can you provide some more information what field you are loading? How this
field looks like? What is the mapping of this field?
If you facet on this field (without executing a script), do you go OOM as
well? What is the expected cardinality of the field you are using in the
script?

As a (slow as hell, so not for production) workaround you could try to use
_source.field instead of doc['field'] - which should prevent the OOM (but
requires a disk seek for each found document).

--Alex

On Fri, Nov 1, 2013 at 1:31 AM, Ben McCann ben@benmccann.com wrote:

I just installed Oracle JDK and am having the same problem. Let me know if
there's anything else I can do to help debug this.

Thanks,
Ben

On Thu, Oct 31, 2013 at 12:16 PM, Ben McCann ben@benmccann.com wrote:

Hi Alex,

I appreciate the response. It sounds like you're suggesting that if I get
an OOM exception once, than all future requests will fail stating there's
been an OOM exception? That's not the case for me. I can run other queries
and only this one fails, so it seems pretty clear the problem is with this
query specifically.

I've attached the log from my elasticsearch server, which has some more
information about the problem. I'm really stuck on this, so I'm hoping your
expertise will get me pointed in the right direction :slight_smile:

Thanks,
Ben

On Thu, Oct 31, 2013 at 3:44 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey Ben,

you ran out of heap/memory. The simplest solution is always to add more
memory to your nodes - ok, simplest solution from an application point of
view :slight_smile:

Obviously it would be great, if we can find out, why the OOM exception
actually happened. Question is, did the exception happen, because you
executed/compiled a script or did it already happen before. You should
check your logs for that.

If it happened before, you should also find out why (facetting on an
analyzed field, sorting on data which does not fit in memory etc). So you
will have to query nodes stats/infos APIs to find more about your running
system.

Another thing is, that you might want to try with an oracle JDK and see
if you get the same problems.

--Alex

On Thu, Oct 31, 2013 at 2:39 AM, Ben McCann <benjamin.j.mccann@gmail.com

wrote:

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though
IllegalAccessError when using an MVEL script filter in the query · Issue #3094 · elastic/elasticsearch · GitHub

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10)
(7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array
for a certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.**matchQuery("myField1",

searchFieldValue1));
query.should(QueryBuilders.**matchQuery("myField2",
searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new

ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].**
values.length");

// Create final scoring function
FunctionScoreQueryBuilder function = QueryBuilders.**functionScoreQuery(query,

script);

This gives me the following errors:

org.elasticsearch.action.**search.**SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][3]:
QueryPhaseExecutionException[[**myIndex][3]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][*myIndex][4]:
ElasticSearchException[tried to access class org.elasticsearch.index.
*fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
nested: IllegalAccessError[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][1]:
QueryPhaseExecutionException[[**myIndex][1]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][2]:
QueryPhaseExecutionException[[**myIndex][2]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][*myIndex][0]:
ElasticSearchException[tried to access class org.elasticsearch.index.
*fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
nested: IllegalAccessError[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
}

Thanks,
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Ben McCann - Mountain View, California, Connectifier, Carnegie Mellon University | about.me

--
Ben McCann - Mountain View, California, Connectifier, Carnegie Mellon University | about.me

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hey Alex,

We have hundreds of millions of documents. Each documents has an array of
objects. I'm trying to boost the score based on the length of that array.
To do so I'm looking at doc['arrayField.objectField'].values.length since
it seems I can't directly look at the array length, but have to look at an
indexed field within that array. I'd guess that arrayField has about three
entries on average and objectField has about 50 chars for each entry.

I'd agree that org.elasticsearch.index.fielddata.
AbstractIndexFieldData.load(AbstractIndexFieldData.java:60) seems to be the
culprit. What data is it trying to load? Is it trying to load the whole
index for the field into memory? How come I have this problem when looking
at the length, but not directly searching against the field? Is it that
searching against the field you can look at the inverted index, but to look
at the array length you have to go the other direction starting with the
document instead?

I'd guess there are two solutions. I can either add more memory to the
machine or index the field length? Any recommendations on which might
be preferable? Can I make a call to elasticsearch to figure out how big the
index is or do I have to estimate it based off what I know about the data?
Can I have elasticsearch store a computed index based on
doc['arrayField.objectField'].values.length
or do I have to update every source document?

Thanks,
Ben

On Fri, Nov 1, 2013 at 7:39 AM, Alexander Reelsen alr@spinscale.de wrote:

Hey,

only quick checking your log, it seems the exception stems from

at
org.elasticsearch.index.fielddata.AbstractIndexFieldData.load(AbstractIndexFieldData.java:60)

which means, it did occur while you tried to load data into memory which
was too big.

Can you provide some more information what field you are loading? How this
field looks like? What is the mapping of this field?
If you facet on this field (without executing a script), do you go OOM as
well? What is the expected cardinality of the field you are using in the
script?

As a (slow as hell, so not for production) workaround you could try to use
_source.field instead of doc['field'] - which should prevent the OOM (but
requires a disk seek for each found document).

--Alex

On Fri, Nov 1, 2013 at 1:31 AM, Ben McCann ben@benmccann.com wrote:

I just installed Oracle JDK and am having the same problem. Let me know
if there's anything else I can do to help debug this.

Thanks,
Ben

On Thu, Oct 31, 2013 at 12:16 PM, Ben McCann ben@benmccann.com wrote:

Hi Alex,

I appreciate the response. It sounds like you're suggesting that if I
get an OOM exception once, than all future requests will fail stating
there's been an OOM exception? That's not the case for me. I can run other
queries and only this one fails, so it seems pretty clear the problem is
with this query specifically.

I've attached the log from my elasticsearch server, which has some more
information about the problem. I'm really stuck on this, so I'm hoping your
expertise will get me pointed in the right direction :slight_smile:

Thanks,
Ben

On Thu, Oct 31, 2013 at 3:44 AM, Alexander Reelsen alr@spinscale.dewrote:

Hey Ben,

you ran out of heap/memory. The simplest solution is always to add more
memory to your nodes - ok, simplest solution from an application point of
view :slight_smile:

Obviously it would be great, if we can find out, why the OOM exception
actually happened. Question is, did the exception happen, because you
executed/compiled a script or did it already happen before. You should
check your logs for that.

If it happened before, you should also find out why (facetting on an
analyzed field, sorting on data which does not fit in memory etc). So you
will have to query nodes stats/infos APIs to find more about your running
system.

Another thing is, that you might want to try with an oracle JDK and see
if you get the same problems.

--Alex

On Thu, Oct 31, 2013 at 2:39 AM, Ben McCann <
benjamin.j.mccann@gmail.com> wrote:

This bug is the only thing I can find that looks at all related. Using
.size() instead of .length doesn't solve the problem for me though
IllegalAccessError when using an MVEL script filter in the query · Issue #3094 · elastic/elasticsearch · GitHub

I'm running Java 7 on Ubuntu 13.04:
$ java -version
java version "1.7.0_25"
OpenJDK Runtime Environment (IcedTea 2.3.10)
(7u25-2.3.10-1ubuntu0.13.04.2)
OpenJDK 64-Bit Server VM (build 23.7-b01, mixed mode)

On Wednesday, October 30, 2013 5:30:11 PM UTC-7, Ben McCann wrote:

I'm trying to boost my query to prefer documents with a longer array
for a certain field, but it's failing to execute.

Any ideas what I might be doing wrong? Here's my code:

// Create query
BoolQueryBuilder query = QueryBuilders.boolQuery();
query.should(QueryBuilders.**matchQuery("myField1",

searchFieldValue1));
query.should(QueryBuilders.**matchQuery("myField2",
searchFieldValue2));

// Boost score based on array length
ScriptScoreFunctionBuilder script = new

ScriptScoreFunctionBuilder();
script.script("_score * doc['myArray.myObjField'].**
values.length");

// Create final scoring function
FunctionScoreQueryBuilder function = QueryBuilders.**functionScoreQuery(query,

script);

This gives me the following errors:

org.elasticsearch.action.**search.**SearchPhaseExecutionException:
Failed to execute phase [query], all shards failed; shardFailures
{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][3]:
QueryPhaseExecutionException[[**myIndex][3]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][4]:
ElasticSearchException[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
nested: IllegalAccessError[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**4033620681383178478830];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][1]:
QueryPhaseExecutionException[[**myIndex][1]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[_score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... _score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][2]:
QueryPhaseExecutionException[[**myIndex][2]: query[filtered(function
score (myField1:searchFieldValue1,**function=script[score *
doc['myArray.myObjField'].values.length], params
[null]))->cache(_type:person)],from[0],size[10]: Query Failed
[Failed to execute main query]]; nested: CompileException[[Error:
java.lang.OutOfMemoryError: Java heap space]
[Near : {... score * doc['myArray.my ....}]
^
[Line: 1, Column: 1]]; nested: ElasticSearchException[java.**lang.OutOfMemoryError:
Java heap space]; nested: ExecutionError[java.lang.**OutOfMemoryError:
Java heap space]; nested: OutOfMemoryError[Java heap space];
}{[Bfi-HZIRQuWIZ6_ApL7sag][**myIndex][0]:
ElasticSearchException[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
nested: IllegalAccessError[tried to access class org.elasticsearch.index.
**fielddata.ScriptDocValues$**Strings$1 from class ASMAccessorImpl**3774454791383178471610];
}

Thanks,
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Ben McCann - Mountain View, California, Connectifier, Carnegie Mellon University | about.me

--
Ben McCann - Mountain View, California, Connectifier, Carnegie Mellon University | about.me

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Yes, for doc access, scripts load all field values into the heap.

I'd not plug endless RAM into machines just because of scripted access to
arrays.

Just index the length of the array alongside with the array and you're fine
by accessing this integer field.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Thanks Jörg. Indexing the array length certainly seems the best option. I
have other advanced scoring I'd like to do, so I want to understand this
well before I run into other trouble. Which operations exactly cause all
values for a field to be loaded into the heap? Can I use a function score
queryhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-function-score-query.htmlat
all without this happening or is it just this one particular operation
of my script that is troublesome?

Thanks,
Ben

On Sat, Nov 2, 2013 at 2:31 AM, joergprante@gmail.com <joergprante@gmail.com

wrote:

Yes, for doc access, scripts load all field values into the heap.

I'd not plug endless RAM into machines just because of scripted access to
arrays.

Just index the length of the array alongside with the array and you're
fine by accessing this integer field.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

I think it is the DocLookup class. In SearchLookup, a synthetic "doc"
variable is handled by DocLookup. From AbstractSearchScript you can see how
doc() is called and an index reader (AtomicReaderContext) is executed to
load all values for a field if you access the field.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/search/lookup/DocLookup.java#L102

It does not matter if you use function score query or not. The crucial
thing is how to obtain field values for a "simple" evaluation like array
length from an inverted index which is not so easy as it would be in a
direct index.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

So it sounds like if I use doc['my_field_name'].value then that would work,
but if I do anything other than access just the field value then it will
load all the values into the cache? Or should I use _fields.my_field_name
or _source.my_field_name? I've seen examples with each, but don't
understand the differences or implications.

Thanks so much for the help!

-Ben

On Sat, Nov 2, 2013 at 6:10 AM, joergprante@gmail.com <joergprante@gmail.com

wrote:

I think it is the DocLookup class. In SearchLookup, a synthetic "doc"
variable is handled by DocLookup. From AbstractSearchScript you can see how
doc() is called and an index reader (AtomicReaderContext) is executed to
load all values for a field if you access the field.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/search/lookup/DocLookup.java#L102

It does not matter if you use function score query or not. The crucial
thing is how to obtain field values for a "simple" evaluation like array
length from an inverted index which is not so easy as it would be in a
direct index.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

The implications are as follows:

_doc (or doc) access by a script loads all the field values of an indexed
field into a cache

_field access by a script accesses a stored field and loads the values of a
field, but not into a cache

_source access by a script loads the whole doc source per call into memory
and does not cache

Therefore the conclusion is

_doc is the fastest but requires most memory, _field is for a stored field,
much slower than _doc, requires not much memory because of not caching, and
_source is the slowest, it loads whole source per doc and requires only a
little memory because of not caching.

The info can be found at

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

13

Thanks! This is super helpful! Last question, I swear... Is the only time
when you would use _source instead of _field when the field is not stored?
In my case for boosting by array length, could I store it as a script
fieldhttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html
and
then access in my function score query as _field to solve the heap overflow
problem I was facing?

Thanks again so much for the awesome help.

-Ben

On Sat, Nov 2, 2013 at 10:11 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

The implications are as follows:

_doc (or doc) access by a script loads all the field values of an indexed
field into a cache

_field access by a script accesses a stored field and loads the values of
a field, but not into a cache

_source access by a script loads the whole doc source per call into memory
and does not cache

Therefore the conclusion is

_doc is the fastest but requires most memory, _field is for a stored
field, much slower than _doc, requires not much memory because of not
caching, and _source is the slowest, it loads whole source per doc and
requires only a little memory because of not caching.

The info can be found at

Elasticsearch Platform — Find real-time answers at scale | Elastic

Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

Unfortunately, script_fields cannot be used for boosting. If so, the values
would also have to be loaded into memory. I don't know another way to
compute an array length of field values in an inverted index. The "JSON
array" is just a "model" for repeatable fields (multi-valued fields) in ES,
they do not exist as an array in the index, so, functions like array length
of field values are becoming very high resource demanding complex functions.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

15

Thanks!

I filed a pull request to update the _source docs with some of the advice
you've given:

I also filed a feature request to allow for storing and indexing a computed
field:

Appreciate all the help!

-Ben

On Sat, Nov 2, 2013 at 1:24 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

Unfortunately, script_fields cannot be used for boosting. If so, the
values would also have to be loaded into memory. I don't know another way
to compute an array length of field values in an inverted index. The "JSON
array" is just a "model" for repeatable fields (multi-valued fields) in ES,
they do not exist as an array in the index, so, functions like array length
of field values are becoming very high resource demanding complex functions.

Jörg

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/SyRV8SS6dDg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.