Switching from mvel to groovy with java methods

Hi,

With ES 1.4, the default scripting language switched from mvel to groovy.
We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth lomig.poyet@gmail.com wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to groovy.
We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEGB4UoNOT8OKo2wC_ezgdgq9_-QMDL7Z5h3mWgq_BNgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Here it is :

[log-2014-02][0]: SearchParseException[[log-2014-02][0]: from[-1],size[0]:

Parse Failure [Failed to parse source
[{"size":0,"aggs":{"prefilter":{"filter":{"and":[{"bool":{"must":[{"term":{"valid":true}},{"term":{"shop_id":"1838"}}]}},{"range":{"date":{"gte":"2014-11-08T23:00:00.000+00:00","lt":"2014-12-09T23:00:00.000+00:00"}}}]},"aggs":{"per_day":{"terms":{"script":"doc.date.date.setZone(org.elasticsearch.common.joda.time.DateTimeZone.forID('Europe/Paris'));doc.date.date.year+'-'+String.format('%02d',doc.date.date.monthOfYear)+'-'+String.format('%02d',doc.date.date.dayOfMonth)","size":31,"order":{"_term":"asc"}},"aggs":{"stats":{"terms":{"field":"type"},"aggs":{"unique":{"filter":{"term":{"unique":true}}}}}}}}}}}]]];
nested:
GroovyScriptCompilationException[MultipleCompilationErrorsException[startup
failed: General error during canonicalization: Method calls not allowed on
[java.lang.String] java.lang.SecurityException: Method calls not allowed on
[java.lang.String] at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitMethodCallExpression(SecureASTCustomizer.java:855)
at
org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:64)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:897)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitExpressionStatement(SecureASTCustomizer.java:777)
at
org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBlockStatement(SecureASTCustomizer.java:737)
at
org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:552)
at
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1047)
at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583)
at
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259) at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245) at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203) at
org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:119)
at
org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475) at
org.elasticsearch.search.aggregations.support.ValuesSourceParser.createScript(ValuesSourceParser.java:193)
at
org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:153)
at
org.elasticsearch.search.aggregations.bucket.terms.TermsParser.parse(TermsParser.java:57)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:130)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:120)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:77)
at
org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 1 error ]];
}{[-Li6K0zKQnW-QBA1Y7xblQ][log-2014-03][0]:
RemoteTransportException[[sql1][inet[/10.16.75.3:9300]][indices:data/read/search[phase/query]]];

Le mercredi 10 décembre 2014 11:06:37 UTC+1, Jörg Prante a écrit :

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth <lomig...@gmail.com
<javascript:>> wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to groovy.
We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2f582608-8840-4602-a1f7-6248fbe2e1fd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

May this exception be caused by a bad dynamic_scripting parameter ?

Le mercredi 10 décembre 2014 11:10:14 UTC+1, Dunaeth a écrit :

Here it is :

[log-2014-02][0]: SearchParseException[[log-2014-02][0]: from[-1],size[0]:

Parse Failure [Failed to parse source
[{"size":0,"aggs":{"prefilter":{"filter":{"and":[{"bool":{"must":[{"term":{"valid":true}},{"term":{"shop_id":"1838"}}]}},{"range":{"date":{"gte":"2014-11-08T23:00:00.000+00:00","lt":"2014-12-09T23:00:00.000+00:00"}}}]},"aggs":{"per_day":{"terms":{"script":"doc.date.date.setZone(org.elasticsearch.common.joda.time.DateTimeZone.forID('Europe/Paris'));doc.date.date.year+'-'+String.format('%02d',doc.date.date.monthOfYear)+'-'+String.format('%02d',doc.date.date.dayOfMonth)","size":31,"order":{"_term":"asc"}},"aggs":{"stats":{"terms":{"field":"type"},"aggs":{"unique":{"filter":{"term":{"unique":true}}}}}}}}}}}]]];
nested:
GroovyScriptCompilationException[MultipleCompilationErrorsException[startup
failed: General error during canonicalization: Method calls not allowed on
[java.lang.String] java.lang.SecurityException: Method calls not allowed on
[java.lang.String] at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitMethodCallExpression(SecureASTCustomizer.java:855)
at
org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:64)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:897)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at
org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitExpressionStatement(SecureASTCustomizer.java:777)
at
org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer$SecuringCodeVisitor.visitBlockStatement(SecureASTCustomizer.java:737)
at
org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
at
org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:552)
at
org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1047)
at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583)
at
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259) at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245) at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203) at
org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:119)
at
org.elasticsearch.script.ScriptService.getCompiledScript(ScriptService.java:353)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475) at
org.elasticsearch.search.aggregations.support.ValuesSourceParser.createScript(ValuesSourceParser.java:193)
at
org.elasticsearch.search.aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:153)
at
org.elasticsearch.search.aggregations.bucket.terms.TermsParser.parse(TermsParser.java:57)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:130)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:120)
at
org.elasticsearch.search.aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:77)
at
org.elasticsearch.search.aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 1 error ]];
}{[-Li6K0zKQnW-QBA1Y7xblQ][log-2014-03][0]:
RemoteTransportException[[sql1][inet[/10.16.75.3:9300]][indices:data/read/search[phase/query]]];

Le mercredi 10 décembre 2014 11:06:37 UTC+1, Jörg Prante a écrit :

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth lomig...@gmail.com wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to
groovy. We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

No. I think ES developers configured the sandbox to deny java.lang.* calls
and java.lang.String is not in the whitelist.

You can add java.lang.String to the whitelist by defining your own
whitelist including java.lang.String

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_groovy_sandboxing

or open an issue with the request to add java.lang.String to the groovy
whitelist by default.

Jörg

On Wed, Dec 10, 2014 at 11:13 AM, Dunaeth lomig.poyet@gmail.com wrote:

May this exception be caused by a bad dynamic_scripting parameter ?

Le mercredi 10 décembre 2014 11:10:14 UTC+1, Dunaeth a écrit :

Here it is :

[log-2014-02][0]: SearchParseException[[log-2014-02][0]:

from[-1],size[0]: Parse Failure [Failed to parse source
[{"size":0,"aggs":{"prefilter":{"filter":{"and":[{"bool":{"
must":[{"term":{"valid":true}},{"term":{"shop_id":"1838"}}]}
},{"range":{"date":{"gte":"2014-11-08T23:00:00.000+00:00"
,"lt":"2014-12-09T23:00:00.000+00:00"}}}]},"aggs":{"per_
day":{"terms":{"script":"doc.date.date.setZone(org.
elasticsearch.common.joda.time.DateTimeZone.forID('
Europe/Paris'));doc.date.date.year+'-'+String.format('%02d',
doc.date.date.monthOfYear)+'-'+String.format('%02d',doc.
date.date.dayOfMonth)","size":31,"order":{"_term":"asc"}},"
aggs":{"stats":{"terms":{"field":"type"},"aggs":{"
unique":{"filter":{"term":{"unique":true}}}}}}}}}}}]]]; nested:
GroovyScriptCompilationException[MultipleCompilationErrorsException[startup
failed: General error during canonicalization: Method calls not allowed on
[java.lang.String] java.lang.SecurityException: Method calls not allowed on
[java.lang.String] at org.codehaus.groovy.control.customizers.
SecureASTCustomizer$SecuringCodeVisitor.visitMethodCallExpression(SecureASTCustomizer.java:855)
at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:64)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:897)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitExpressionStatement(SecureASTCustomizer.java:777)
at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBlockStatement(SecureASTCustomizer.java:737)
at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:552)
at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1047)
at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203)
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(
GroovyScriptEngineService.java:119) at org.elasticsearch.script.
ScriptService.getCompiledScript(ScriptService.java:353) at
org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.aggregations.support.ValuesSourceParser.
createScript(ValuesSourceParser.java:193) at org.elasticsearch.search.
aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:153)
at org.elasticsearch.search.aggregations.bucket.terms.
TermsParser.parse(TermsParser.java:57) at org.elasticsearch.search.
aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:130)
at org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:120) at
org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:77) at org.elasticsearch.search.
aggregations.AggregationParseElement.parse(AggregationParseElement.java:60)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:665)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231) at org.elasticsearch.search.
action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559) at java.util.concurrent.
ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 1 error ]];
}{[-Li6K0zKQnW-QBA1Y7xblQ][log-2014-03][0]: RemoteTransportException[[
sql1][inet[/10.16.75.3:9300]][indices:data/read/search[phase/query]]];

Le mercredi 10 décembre 2014 11:06:37 UTC+1, Jörg Prante a écrit :

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth lomig...@gmail.com wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to
groovy. We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHV%3D3%3DKUQPM2-KC0OJCvypedhuvZTFNzgHeQzRZjADgJw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks, I'll just need to find what is the actual whitelist and how to have
a custom one then. If someone has any clue ?

Le mercredi 10 décembre 2014 11:27:52 UTC+1, Jörg Prante a écrit :

No. I think ES developers configured the sandbox to deny java.lang.* calls
and java.lang.String is not in the whitelist.

https://github.com/elasticsearch/elasticsearch/blob/b43b56a6a85f7dd131086fd83dc9267aecbbf0a3/src/main/java/org/elasticsearch/script/groovy/GroovySandboxExpressionChecker.java#L90-L111
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fblob%2Fb43b56a6a85f7dd131086fd83dc9267aecbbf0a3%2Fsrc%2Fmain%2Fjava%2Forg%2Felasticsearch%2Fscript%2Fgroovy%2FGroovySandboxExpressionChecker.java%23L90-L111&sa=D&sntz=1&usg=AFQjCNHRcEKxmTeoBh0RSXOdklvMO2vAbg

You can add java.lang.String to the whitelist by defining your own
whitelist including java.lang.String

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_groovy_sandboxing

or open an issue with the request to add java.lang.String to the groovy
whitelist by default.

Jörg

On Wed, Dec 10, 2014 at 11:13 AM, Dunaeth <lomig...@gmail.com
<javascript:>> wrote:

May this exception be caused by a bad dynamic_scripting parameter ?

Le mercredi 10 décembre 2014 11:10:14 UTC+1, Dunaeth a écrit :

Here it is :

[log-2014-02][0]: SearchParseException[[log-2014-02][0]:

from[-1],size[0]: Parse Failure [Failed to parse source
[{"size":0,"aggs":{"prefilter":{"filter":{"and":[{"bool":{"
must":[{"term":{"valid":true}},{"term":{"shop_id":"1838"}}]}
},{"range":{"date":{"gte":"2014-11-08T23:00:00.000+00:00"
,"lt":"2014-12-09T23:00:00.000+00:00"}}}]},"aggs":{"per_
day":{"terms":{"script":"doc.date.date.setZone(org.
elasticsearch.common.joda.time.DateTimeZone.forID('
Europe/Paris'));doc.date.date.year+'-'+String.format('%02d',
doc.date.date.monthOfYear)+'-'+String.format('%02d',doc.
date.date.dayOfMonth)","size":31,"order":{"_term":"asc"}},"
aggs":{"stats":{"terms":{"field":"type"},"aggs":{"
unique":{"filter":{"term":{"unique":true}}}}}}}}}}}]]]; nested:
GroovyScriptCompilationException[MultipleCompilationErrorsException[startup
failed: General error during canonicalization: Method calls not allowed on
[java.lang.String] java.lang.SecurityException: Method calls not allowed on
[java.lang.String] at org.codehaus.groovy.control.customizers.
SecureASTCustomizer$SecuringCodeVisitor.visitMethodCallExpression(SecureASTCustomizer.java:855)
at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:64)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:897)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitExpressionStatement(SecureASTCustomizer.java:777)
at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBlockStatement(SecureASTCustomizer.java:737)
at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:552)
at org.codehaus.groovy.control.CompilationUnit.
applyToPrimaryClassNodes(CompilationUnit.java:1047) at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203)
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(
GroovyScriptEngineService.java:119) at org.elasticsearch.script.
ScriptService.getCompiledScript(ScriptService.java:353) at
org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.aggregations.support.ValuesSourceParser.
createScript(ValuesSourceParser.java:193) at org.elasticsearch.search.
aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:153)
at org.elasticsearch.search.aggregations.bucket.terms.
TermsParser.parse(TermsParser.java:57) at org.elasticsearch.search.
aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:130)
at org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:120) at
org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:77) at
org.elasticsearch.search.aggregations.AggregationParseElement.parse(
AggregationParseElement.java:60) at org.elasticsearch.search.
SearchService.parseSource(SearchService.java:665) at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231) at org.elasticsearch.search.
action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228) at org.elasticsearch.search.
action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559) at java.util.concurrent.
ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 1 error ]];
}{[-Li6K0zKQnW-QBA1Y7xblQ][log-2014-03][0]: RemoteTransportException[[
sql1][inet[/10.16.75.3:9300]][indices:data/read/search[phase/query]]];

Le mercredi 10 décembre 2014 11:06:37 UTC+1, Jörg Prante a écrit :

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth lomig...@gmail.com wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to
groovy. We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e87161b5-dd84-4a38-8dfd-47bd765133be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I figured out what the actual default receiver_whitelist is through the
GroovySandboxExpressionChecked code, and if I want to add java.lang.String
to the whitelist, I'd just have to add its classname to the other
classnames in the script.groovy.sandbox.receiver_whitelist setting in
elasticsearch.yml
If I'm not wrong, it should be :

script.groovy.sandbox.receiver_whitelist:
"java.lang.Math,java.lang.Integer,java.lang.Float,java.lang.Double,java.lang.Long,java.lang.Short,java.lang.Character,java.lang.Byte,java.lang.Boolean,java.math.BigDecimal,java.util.Arrays,java.util.Date,java.util.List,java.util.Map,java.util.Set,java.lang.Object,org.elasticsearch.common.joda.time.DateTime,org.elasticsearch.common.joda.time.DateTimeUtils,org.elasticsearch.common.joda.time.DateTimeZone,org.elasticsearch.common.joda.time.Instant,java.lang.String"

am I right ?

Le mercredi 10 décembre 2014 11:32:45 UTC+1, Dunaeth a écrit :

Thanks, I'll just need to find what is the actual whitelist and how to
have a custom one then. If someone has any clue ?

Le mercredi 10 décembre 2014 11:27:52 UTC+1, Jörg Prante a écrit :

No. I think ES developers configured the sandbox to deny java.lang.*
calls and java.lang.String is not in the whitelist.

https://github.com/elasticsearch/elasticsearch/blob/b43b56a6a85f7dd131086fd83dc9267aecbbf0a3/src/main/java/org/elasticsearch/script/groovy/GroovySandboxExpressionChecker.java#L90-L111
https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Felasticsearch%2Felasticsearch%2Fblob%2Fb43b56a6a85f7dd131086fd83dc9267aecbbf0a3%2Fsrc%2Fmain%2Fjava%2Forg%2Felasticsearch%2Fscript%2Fgroovy%2FGroovySandboxExpressionChecker.java%23L90-L111&sa=D&sntz=1&usg=AFQjCNHRcEKxmTeoBh0RSXOdklvMO2vAbg

You can add java.lang.String to the whitelist by defining your own
whitelist including java.lang.String

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html#_groovy_sandboxing

or open an issue with the request to add java.lang.String to the groovy
whitelist by default.

Jörg

On Wed, Dec 10, 2014 at 11:13 AM, Dunaeth lomig...@gmail.com wrote:

May this exception be caused by a bad dynamic_scripting parameter ?

Le mercredi 10 décembre 2014 11:10:14 UTC+1, Dunaeth a écrit :

Here it is :

[log-2014-02][0]: SearchParseException[[log-2014-02][0]:

from[-1],size[0]: Parse Failure [Failed to parse source
[{"size":0,"aggs":{"prefilter":{"filter":{"and":[{"bool":{"
must":[{"term":{"valid":true}},{"term":{"shop_id":"1838"}}]}
},{"range":{"date":{"gte":"2014-11-08T23:00:00.000+00:00"
,"lt":"2014-12-09T23:00:00.000+00:00"}}}]},"aggs":{"per_
day":{"terms":{"script":"doc.date.date.setZone(org.
elasticsearch.common.joda.time.DateTimeZone.forID('
Europe/Paris'));doc.date.date.year+'-'+String.format('%02d',
doc.date.date.monthOfYear)+'-'+String.format('%02d',doc.
date.date.dayOfMonth)","size":31,"order":{"_term":"asc"}},"
aggs":{"stats":{"terms":{"field":"type"},"aggs":{"
unique":{"filter":{"term":{"unique":true}}}}}}}}}}}]]]; nested:
GroovyScriptCompilationException[MultipleCompilationErrorsException[startup
failed: General error during canonicalization: Method calls not allowed on
[java.lang.String] java.lang.SecurityException: Method calls not allowed on
[java.lang.String] at org.codehaus.groovy.control.customizers.
SecureASTCustomizer$SecuringCodeVisitor.visitMethodCallExpression(SecureASTCustomizer.java:855)
at org.codehaus.groovy.ast.expr.MethodCallExpression.visit(MethodCallExpression.java:64)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:897)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBinaryExpression(SecureASTCustomizer.java:896)
at org.codehaus.groovy.ast.expr.BinaryExpression.visit(BinaryExpression.java:49)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitExpressionStatement(SecureASTCustomizer.java:777)
at org.codehaus.groovy.ast.stmt.ExpressionStatement.visit(ExpressionStatement.java:40)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer$
SecuringCodeVisitor.visitBlockStatement(SecureASTCustomizer.java:737)
at org.codehaus.groovy.ast.stmt.BlockStatement.visit(BlockStatement.java:69)
at org.codehaus.groovy.control.customizers.SecureASTCustomizer.call(SecureASTCustomizer.java:552)
at org.codehaus.groovy.control.CompilationUnit.
applyToPrimaryClassNodes(CompilationUnit.java:1047) at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:583)
at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:561)
at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:538)
at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:286)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:259)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:245)
at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:203)
at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(
GroovyScriptEngineService.java:119) at org.elasticsearch.script.
ScriptService.getCompiledScript(ScriptService.java:353) at
org.elasticsearch.script.ScriptService.compile(ScriptService.java:339)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:475)
at org.elasticsearch.search.aggregations.support.ValuesSourceParser.
createScript(ValuesSourceParser.java:193) at org.elasticsearch.search.
aggregations.support.ValuesSourceParser.config(ValuesSourceParser.java:153)
at org.elasticsearch.search.aggregations.bucket.terms.
TermsParser.parse(TermsParser.java:57) at org.elasticsearch.search.
aggregations.AggregatorParsers.parseAggregators(AggregatorParsers.java:130)
at org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:120) at
org.elasticsearch.search.aggregations.AggregatorParsers.
parseAggregators(AggregatorParsers.java:77) at
org.elasticsearch.search.aggregations.AggregationParseElement.parse(
AggregationParseElement.java:60) at org.elasticsearch.search.
SearchService.parseSource(SearchService.java:665) at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:231) at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228) at org.elasticsearch.search.
action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559) at java.util.concurrent.
ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745) 1 error ]];
}{[-Li6K0zKQnW-QBA1Y7xblQ][log-2014-03][0]: RemoteTransportException[[
sql1][inet[/10.16.75.3:9300]][indices:data/read/search[phase/query]]];

Le mercredi 10 décembre 2014 11:06:37 UTC+1, Jörg Prante a écrit :

Can you post the security exception?

Jörg

On Wed, Dec 10, 2014 at 11:02 AM, Dunaeth lomig...@gmail.com wrote:

Hi,

With ES 1.4, the default scripting language switched from mvel to
groovy. We were using script fields in our queries like
«String.format('%02d',doc.date.date.monthOfYear)», is there a way to
achieve the same result using groovy ? ATM, we're experiencing security
exceptions, method calls not allowed.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/d9afcc9a-f4a5-411f-9fd2-0c51f44a5f2a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce3d5233-d0cb-41a5-8797-92fa3275fb26%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4772fb46-74cf-438d-ab2f-19c4eacdc995%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.