# Intermittent VerifyError in update/upsert script with nested documents

**URL:** <https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909>\
**Category:** Elasticsearch\
**Created:** [May 10, 2013, 1:53am UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909 "2013-05-10T01:53:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rockbobsta](https://avatars.discourse-cdn.com/v4/letter/r/90ced4/32.png) [@rockbobsta](https://discuss.elastic.co/u/rockbobsta)\
**Post date:** [May 10, 2013, 1:53am UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909/1 "2013-05-10T01:53:32Z")

</div>

Hi,

I'm having an intermittent problem with an upsert statement. The issue  
appears to be in compiling the script, but only occurs under particular  
circumstances.  
I've seen two similar issues reported, but I believe this is slightly  
different as it's using the standard script, not an MVEL script.  
These reports mention a very similar stacktrace to what I'm seeing:  
[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ)  
[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ)

I've put together a quick gist to reproduce the problem:

> <https://gist.github.com/baxford/5551877>

Here are some more notes:

- I can reliably reproduce the problem if I create the index, insert the 2  
records in the gist, then stop elasticsearch, restart it, and try to  
execute the upsert statement.
- When the VerifyError occurs, the CURL request just hangs.
- Interestingly, it appears to happen only on a document with a large  
number of nested docs, while it doesn't appear to happen with a doc  
containing a few nested docs
- Even for the upsert that causes the VerifyError the first time, if I  
cancel the CURL request then run it again, it works properly.

The main reason this is an issue is that we are running similar upsert  
statements as this within and elasticsearch river. When the VerifyError  
occurs, the whole thread seems to block indefinitely and the river stops  
processing anything.

It's possible as suggested in other posts, that it's a bug in MVEL, but it  
seems that i'm not using MVEL for my scripts (although possibly the script  
I'm using gets compiled to MVEL at a later time?).  
Here are some other details:  
I first noticed this issue on a production AWS machine running:

- Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86\_64)
- java version "1.7.0\_17"
- Elasticsearch 0.20.6

To reproduce it, I'm running this locally on

- mac osx 10.7.5,
- java version "1.6.0\_43"
- latest master version of Elasticsearch from github (I'm using the latest  
from github in case it's been fixed or is using a more recent version of  
MVEL which fixes the problem)

Any assistance would be appreciated. I'll continue to debug and see if I  
can get any more info on the cause or a fix.  
thanks.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [May 10, 2013, 11:43am UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909/2 "2013-05-10T11:43:00Z")

</div>

Hey

thanks a lot for taking your time to create a reproducable report. I've  
created an issue at

> <https://github.com/elastic/elasticsearch/issues/3021>
>
> Original report came from the mailing list at
> 
> https://groups.google.com/forum/?…fromgroups=#!searchin/elasticsearch/verifyerror/elasticsearch/noCI7fBRsHw/t4rSML4D8jcJ
> 
> I'll put the gist here for readability (orig at https://gist.github.com/baxford/5551877)
> 
> \`\`\`
> curl -XDELETE 'http://localhost:9200/app'
> curl -XPUT 'http://localhost:9200/app' -d '
> {
> "mappings" : {
> "parent": {
> "properties" : {
> "children": {
> "type" : "nested",
> "properties" : {
> "id" : {"store" : false, "type": "long", "include\_in\_all": false},
> "sent" : {"store" : false, "type": "long", "include\_in\_all": false}
> }
> }
> }
> }               
> }
> }'
> 
> curl -XPUT 'http://localhost:9200/app/parent/2nested' -d '
> {
> "children": \[
> {
> "id": 1967,
> "sent": 1367561477819
> },
> {
> "id": 1968,
> "sent": 1367723988849
> }
> \]
> }
> '
> curl -XPUT 'http://localhost:9200/app/parent/manyNested' -d '
> {"children":\[{"id":1970,"sent":1367557829592},{"id":1967,"sent":1367561477651},{"id":1963,"sent":1367566046098},{"id":2220,"sent":1367574093902},{"id":2221,"sent":1367574178118},{"id":1949,"sent":1367577441224},{"id":2230,"sent":1367578681929},{"id":2255,"sent":1367582445865},{"id":2229,"sent":1367582978773},{"id":2036,"sent":1367586060714},{"id":2181,"sent":1367588265555},{"id":2135,"sent":1367589621826},{"id":2040,"sent":1367590988082},{"id":2265,"sent":1367597111092},{"id":1961,"sent":1367597766420},{"id":2368,"sent":1367607084937},{"id":2325,"sent":1367604622108},{"id":2380,"sent":1367632844989},{"id":2449,"sent":1367637386597},{"id":2162,"sent":1367639335040},{"id":2457,"sent":1367640147955},{"id":2359,"sent":1367640976257},{"id":1995,"sent":1367642443645},{"id":2106,"sent":1367644004789},{"id":2483,"sent":1367645257597},{"id":2244,"sent":1367643998849},{"id":2070,"sent":1367645763061},{"id":1955,"sent":1367648027479},{"id":2413,"sent":1367647150696},{"id":2432,"sent":1367644680952},{"id":2236,"sent":1367645491674},{"id":2154,"sent":1367648593594},{"id":2017,"sent":1367645542935},{"id":2437,"sent":1367649600428},{"id":2447,"sent":1367648433362},{"id":2115,"sent":1367650722908},{"id":2147,"sent":1367649384788},{"id":2415,"sent":1367652386667},{"id":2490,"sent":1367660500351},{"id":2431,"sent":1367660240898},{"id":2565,"sent":1367661020928},{"id":2501,"sent":1367661971510},{"id":2078,"sent":1367663056515},{"id":2184,"sent":1367662922221},{"id":2583,"sent":1367664149452},{"id":2195,"sent":1367664839517},{"id":2562,"sent":1367664317481},{"id":2286,"sent":1367664384121},{"id":2462,"sent":1367667431564},{"id":2332,"sent":1367669188915},{"id":2143,"sent":1367674582587},{"id":1957,"sent":1367675023621},{"id":2238,"sent":1367685309996},{"id":2074,"sent":1367687406146},{"id":2374,"sent":1367686493566},{"id":2336,"sent":1367693189820},{"id":2317,"sent":1367693900534},{"id":2095,"sent":1367695156486},{"id":1947,"sent":1367699112244},{"id":2031,"sent":1367699831117},{"id":2356,"sent":1367699040377},{"id":2492,"sent":1367699156344},{"id":2407,"sent":1367688683169},{"id":2122,"sent":1367699317524},{"id":2049,"sent":1367699998534},{"id":2352,"sent":1367705893143},{"id":2611,"sent":1367704332534},{"id":2463,"sent":1367705247247},{"id":2295,"sent":1367705743731},{"id":2428,"sent":1367707924514},{"id":2713,"sent":1367711489640},{"id":1998,"sent":1367688901861},{"id":2218,"sent":1367715074441},{"id":2438,"sent":1367688967128},{"id":2668,"sent":1367715433513},{"id":2429,"sent":1367716384220},{"id":2081,"sent":1367717543664},{"id":2232,"sent":1367717580481},{"id":2405,"sent":1367718543453},{"id":2133,"sent":1367702124091},{"id":2185,"sent":1367709922527},{"id":2053,"sent":1367720072185},{"id":2409,"sent":1367711905080},{"id":2489,"sent":1367712328854},{"id":2402,"sent":1367713280879},{"id":2007,"sent":1367714124287},{"id":2704,"sent":1367714511892},{"id":2388,"sent":1367715713018},{"id":2360,"sent":1367716008115},{"id":2004,"sent":1367721618552},{"id":2508,"sent":1367717122370},{"id":2027,"sent":1367717322516},{"id":2502,"sent":1367721633478},{"id":2697,"sent":1367718063199},{"id":2525,"sent":1367722480411},{"id":1974,"sent":1367722629388},{"id":2165,"sent":1367718180288},{"id":2695,"sent":1367718702869},{"id":1976,"sent":1367719463806},{"id":1968,"sent":1367721068881},{"id":2030,"sent":1367722280029},{"id":2300,"sent":1367725603161}\]}
> '
> 
> \# Stopping ES, restarting it, then executing the following seems to cause the VerifyError in script compilation 
> \# TransportUpdateActiong.shardOperation, line 310
> \# when executing script.run();
> curl -XPOST 'http://localhost:9200/app/parent/manyNested/\_update' -d '{
> "script" : 
> "if(ctx.\_source\[\\"children\\"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\[\\"children\\"\] == null) { ctx.\_source.children = \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] } else {ctx.\_source.children += \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] }"
> ,
> "params" : {
> "parent\_id" : "1183146090417",
> "timestamp" : 1367722894963,
> "child\_id" : 2030
> },
> "upsert" : {
> "children": \[
> {
> "id": 2030,
> "sent": 1367722894963
> }
> \]
> }
> }' && echo
> 
> \# However, the same issue doesn't seem to happen to a doc with less nested docs
> curl -XPOST 'http://localhost:9200/app/parent/2nested/\_update' -d '{
> "script" : 
> "if(ctx.\_source\[\\"children\\"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\[\\"children\\"\] == null) { ctx.\_source.children = \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] } else {ctx.\_source.children += \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] }"
> ,
> "params" : {
> "parent\_id" : "1183146090417",
> "timestamp" : 1367722894963,
> "child\_id" : 2030
> },
> "upsert" : {
> "children": \[
> {
> "id": 2030,
> "sent": 1367722894963
> }
> \]
> }
> }' && echo
> \`\`\`
> 
> Haven't taken a deeper look at it, but can confirm the curl call hangs and there is this exception in the logs:
> 
> \`\`\`
> \*\*\*\* COMPILER BUG! REPORT THIS IMMEDIATELY AT http://jira.codehaus.org/browse/mvel2
> Expression: if(ctx.\_source\["children"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\["children"\] == null) { ctx.\_source.children = \[{"id": child\_id, "sent": timestamp }\] } else {ctx.\_source.children += \[{"id": child\_id, "sent": timestamp }\] }
> Exception in thread "elasticsearch\[Acrobat\]\[index\]\[T#1\]" java.lang.VerifyError: (class: ASMAccessorImpl\_10448523421368185357780, method: getValue signature: (Ljava/lang/Object;Ljava/lang/Object;Lorg/elasticsearch/common/mvel2/integration/VariableResolverFactory;)Ljava/lang/Object;) Expecting to find integer on stack
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2398)
> at java.lang.Class.getConstructor0(Class.java:2708)
> at java.lang.Class.newInstance0(Class.java:328)
> at java.lang.Class.newInstance(Class.java:310)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.\_initializeAccessor(ASMAccessorOptimizer.java:725)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.compileAccessor(ASMAccessorOptimizer.java:859)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.optimizeAccessor(ASMAccessorOptimizer.java:243)
> at org.elasticsearch.common.mvel2.optimizers.dynamic.DynamicGetAccessor.optimize(DynamicGetAccessor.java:90)
> at org.elasticsearch.common.mvel2.optimizers.dynamic.DynamicGetAccessor.getValue(DynamicGetAccessor.java:64)
> at org.elasticsearch.common.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108)
> at org.elasticsearch.common.mvel2.ast.BinaryOperation.getReducedValueAccelerated(BinaryOperation.java:108)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.IfNode.getReducedValueAccelerated(IfNode.java:73)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.ForNode.getReducedValueAccelerated(ForNode.java:67)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.IfNode.getReducedValueAccelerated(IfNode.java:74)
> at org.elasticsearch.common.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113)
> at org.elasticsearch.script.mvel.MvelScriptEngineService$MvelExecutableScript.run(MvelScriptEngineService.java:130)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:310)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:211)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:85)
> at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:191)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> \`\`\`

Please add anything important there, if you find out more or I described  
something incorrectly. Thanks a lot!

--Alex

On Fri, May 10, 2013 at 3:53 AM, rockbobsta [bob@figjamit.com.au](mailto:bob@figjamit.com.au) wrote:

> Hi,
> 
> I'm having an intermittent problem with an upsert statement. The issue  
> appears to be in compiling the script, but only occurs under particular  
> circumstances.  
> I've seen two similar issues reported, but I believe this is slightly  
> different as it's using the standard script, not an MVEL script.  
> These reports mention a very similar stacktrace to what I'm seeing:
> 
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ)
> 
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ)
> 
> I've put together a quick gist to reproduce the problem:  
> [Replicate VerifyError in ElasticSearch update/upsert script · GitHub](https://gist.github.com/baxford/5551877)
> 
> Here are some more notes:
> 
> - I can reliably reproduce the problem if I create the index, insert the 2  
> records in the gist, then stop elasticsearch, restart it, and try to  
> execute the upsert statement.
> - When the VerifyError occurs, the CURL request just hangs.
> - Interestingly, it appears to happen only on a document with a large  
> number of nested docs, while it doesn't appear to happen with a doc  
> containing a few nested docs
> - Even for the upsert that causes the VerifyError the first time, if I  
> cancel the CURL request then run it again, it works properly.
> 
> The main reason this is an issue is that we are running similar upsert  
> statements as this within and elasticsearch river. When the VerifyError  
> occurs, the whole thread seems to block indefinitely and the river stops  
> processing anything.
> 
> It's possible as suggested in other posts, that it's a bug in MVEL, but it  
> seems that i'm not using MVEL for my scripts (although possibly the script  
> I'm using gets compiled to MVEL at a later time?).  
> Here are some other details:  
> I first noticed this issue on a production AWS machine running:
> 
> - Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86\_64)
> - java version "1.7.0\_17"
> - Elasticsearch 0.20.6
> 
> To reproduce it, I'm running this locally on
> 
> - mac osx 10.7.5,
> - java version "1.6.0\_43"
> - latest master version of Elasticsearch from github (I'm using the latest  
> from github in case it's been fixed or is using a more recent version of  
> MVEL which fixes the problem)
> 
> Any assistance would be appreciated. I'll continue to debug and see if I  
> can get any more info on the cause or a fix.  
> thanks.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![rockbobsta](https://avatars.discourse-cdn.com/v4/letter/r/90ced4/32.png) [@rockbobsta](https://discuss.elastic.co/u/rockbobsta)\
**Post date:** [May 12, 2013, 11:26pm UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909/3 "2013-05-12T23:26:33Z")

</div>

Hi Alex,

no probs, I'll update if I find out any more.  
thanks for the quick response!

regards  
Bob.

On Friday, May 10, 2013 9:43:00 PM UTC+10, Alexander Reelsen wrote:

> Hey
> 
> thanks a lot for taking your time to create a reproducable report. I've  
> created an issue at  
> [MVEL VerifyError with update API after restarting elasticsearch · Issue #3021 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3021)  
> Please add anything important there, if you find out more or I described  
> something incorrectly. Thanks a lot!
> 
> --Alex
> 
> On Fri, May 10, 2013 at 3:53 AM, rockbobsta \<b...@figjamit.com.au\<javascript:\>
> 
> > wrote:
> 
> > Hi,
> > 
> > I'm having an intermittent problem with an upsert statement. The issue  
> > appears to be in compiling the script, but only occurs under particular  
> > circumstances.  
> > I've seen two similar issues reported, but I believe this is slightly  
> > different as it's using the standard script, not an MVEL script.  
> > These reports mention a very similar stacktrace to what I'm seeing:
> > 
> > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ)
> > 
> > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ)
> > 
> > I've put together a quick gist to reproduce the problem:  
> > [Replicate VerifyError in ElasticSearch update/upsert script · GitHub](https://gist.github.com/baxford/5551877)
> > 
> > Here are some more notes:
> > 
> > - I can reliably reproduce the problem if I create the index, insert the  
> > 2 records in the gist, then stop elasticsearch, restart it, and try to  
> > execute the upsert statement.
> > - When the VerifyError occurs, the CURL request just hangs.
> > - Interestingly, it appears to happen only on a document with a large  
> > number of nested docs, while it doesn't appear to happen with a doc  
> > containing a few nested docs
> > - Even for the upsert that causes the VerifyError the first time, if I  
> > cancel the CURL request then run it again, it works properly.
> > 
> > The main reason this is an issue is that we are running similar upsert  
> > statements as this within and elasticsearch river. When the VerifyError  
> > occurs, the whole thread seems to block indefinitely and the river stops  
> > processing anything.
> > 
> > It's possible as suggested in other posts, that it's a bug in MVEL, but  
> > it seems that i'm not using MVEL for my scripts (although possibly the  
> > script I'm using gets compiled to MVEL at a later time?).  
> > Here are some other details:  
> > I first noticed this issue on a production AWS machine running:
> > 
> > - Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86\_64)
> > - java version "1.7.0\_17"
> > - Elasticsearch 0.20.6
> > 
> > To reproduce it, I'm running this locally on
> > 
> > - mac osx 10.7.5,
> > - java version "1.6.0\_43"
> > - latest master version of Elasticsearch from github (I'm using the  
> > latest from github in case it's been fixed or is using a more recent  
> > version of MVEL which fixes the problem)
> > 
> > Any assistance would be appreciated. I'll continue to debug and see if I  
> > can get any more info on the cause or a fix.  
> > thanks.
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [May 13, 2013, 6:15am UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909/4 "2013-05-13T06:15:11Z")

</div>

Hey,

Igor commented in the ticket meanwhile, maybe it helps you... at least to  
switch from MVEL to javascript or similar 🙂

> <https://github.com/elastic/elasticsearch/issues/3021>
>
> Original report came from the mailing list at
> 
> https://groups.google.com/forum/?…fromgroups=#!searchin/elasticsearch/verifyerror/elasticsearch/noCI7fBRsHw/t4rSML4D8jcJ
> 
> I'll put the gist here for readability (orig at https://gist.github.com/baxford/5551877)
> 
> \`\`\`
> curl -XDELETE 'http://localhost:9200/app'
> curl -XPUT 'http://localhost:9200/app' -d '
> {
> "mappings" : {
> "parent": {
> "properties" : {
> "children": {
> "type" : "nested",
> "properties" : {
> "id" : {"store" : false, "type": "long", "include\_in\_all": false},
> "sent" : {"store" : false, "type": "long", "include\_in\_all": false}
> }
> }
> }
> }               
> }
> }'
> 
> curl -XPUT 'http://localhost:9200/app/parent/2nested' -d '
> {
> "children": \[
> {
> "id": 1967,
> "sent": 1367561477819
> },
> {
> "id": 1968,
> "sent": 1367723988849
> }
> \]
> }
> '
> curl -XPUT 'http://localhost:9200/app/parent/manyNested' -d '
> {"children":\[{"id":1970,"sent":1367557829592},{"id":1967,"sent":1367561477651},{"id":1963,"sent":1367566046098},{"id":2220,"sent":1367574093902},{"id":2221,"sent":1367574178118},{"id":1949,"sent":1367577441224},{"id":2230,"sent":1367578681929},{"id":2255,"sent":1367582445865},{"id":2229,"sent":1367582978773},{"id":2036,"sent":1367586060714},{"id":2181,"sent":1367588265555},{"id":2135,"sent":1367589621826},{"id":2040,"sent":1367590988082},{"id":2265,"sent":1367597111092},{"id":1961,"sent":1367597766420},{"id":2368,"sent":1367607084937},{"id":2325,"sent":1367604622108},{"id":2380,"sent":1367632844989},{"id":2449,"sent":1367637386597},{"id":2162,"sent":1367639335040},{"id":2457,"sent":1367640147955},{"id":2359,"sent":1367640976257},{"id":1995,"sent":1367642443645},{"id":2106,"sent":1367644004789},{"id":2483,"sent":1367645257597},{"id":2244,"sent":1367643998849},{"id":2070,"sent":1367645763061},{"id":1955,"sent":1367648027479},{"id":2413,"sent":1367647150696},{"id":2432,"sent":1367644680952},{"id":2236,"sent":1367645491674},{"id":2154,"sent":1367648593594},{"id":2017,"sent":1367645542935},{"id":2437,"sent":1367649600428},{"id":2447,"sent":1367648433362},{"id":2115,"sent":1367650722908},{"id":2147,"sent":1367649384788},{"id":2415,"sent":1367652386667},{"id":2490,"sent":1367660500351},{"id":2431,"sent":1367660240898},{"id":2565,"sent":1367661020928},{"id":2501,"sent":1367661971510},{"id":2078,"sent":1367663056515},{"id":2184,"sent":1367662922221},{"id":2583,"sent":1367664149452},{"id":2195,"sent":1367664839517},{"id":2562,"sent":1367664317481},{"id":2286,"sent":1367664384121},{"id":2462,"sent":1367667431564},{"id":2332,"sent":1367669188915},{"id":2143,"sent":1367674582587},{"id":1957,"sent":1367675023621},{"id":2238,"sent":1367685309996},{"id":2074,"sent":1367687406146},{"id":2374,"sent":1367686493566},{"id":2336,"sent":1367693189820},{"id":2317,"sent":1367693900534},{"id":2095,"sent":1367695156486},{"id":1947,"sent":1367699112244},{"id":2031,"sent":1367699831117},{"id":2356,"sent":1367699040377},{"id":2492,"sent":1367699156344},{"id":2407,"sent":1367688683169},{"id":2122,"sent":1367699317524},{"id":2049,"sent":1367699998534},{"id":2352,"sent":1367705893143},{"id":2611,"sent":1367704332534},{"id":2463,"sent":1367705247247},{"id":2295,"sent":1367705743731},{"id":2428,"sent":1367707924514},{"id":2713,"sent":1367711489640},{"id":1998,"sent":1367688901861},{"id":2218,"sent":1367715074441},{"id":2438,"sent":1367688967128},{"id":2668,"sent":1367715433513},{"id":2429,"sent":1367716384220},{"id":2081,"sent":1367717543664},{"id":2232,"sent":1367717580481},{"id":2405,"sent":1367718543453},{"id":2133,"sent":1367702124091},{"id":2185,"sent":1367709922527},{"id":2053,"sent":1367720072185},{"id":2409,"sent":1367711905080},{"id":2489,"sent":1367712328854},{"id":2402,"sent":1367713280879},{"id":2007,"sent":1367714124287},{"id":2704,"sent":1367714511892},{"id":2388,"sent":1367715713018},{"id":2360,"sent":1367716008115},{"id":2004,"sent":1367721618552},{"id":2508,"sent":1367717122370},{"id":2027,"sent":1367717322516},{"id":2502,"sent":1367721633478},{"id":2697,"sent":1367718063199},{"id":2525,"sent":1367722480411},{"id":1974,"sent":1367722629388},{"id":2165,"sent":1367718180288},{"id":2695,"sent":1367718702869},{"id":1976,"sent":1367719463806},{"id":1968,"sent":1367721068881},{"id":2030,"sent":1367722280029},{"id":2300,"sent":1367725603161}\]}
> '
> 
> \# Stopping ES, restarting it, then executing the following seems to cause the VerifyError in script compilation 
> \# TransportUpdateActiong.shardOperation, line 310
> \# when executing script.run();
> curl -XPOST 'http://localhost:9200/app/parent/manyNested/\_update' -d '{
> "script" : 
> "if(ctx.\_source\[\\"children\\"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\[\\"children\\"\] == null) { ctx.\_source.children = \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] } else {ctx.\_source.children += \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] }"
> ,
> "params" : {
> "parent\_id" : "1183146090417",
> "timestamp" : 1367722894963,
> "child\_id" : 2030
> },
> "upsert" : {
> "children": \[
> {
> "id": 2030,
> "sent": 1367722894963
> }
> \]
> }
> }' && echo
> 
> \# However, the same issue doesn't seem to happen to a doc with less nested docs
> curl -XPOST 'http://localhost:9200/app/parent/2nested/\_update' -d '{
> "script" : 
> "if(ctx.\_source\[\\"children\\"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\[\\"children\\"\] == null) { ctx.\_source.children = \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] } else {ctx.\_source.children += \[{\\"id\\": child\_id, \\"sent\\": timestamp }\] }"
> ,
> "params" : {
> "parent\_id" : "1183146090417",
> "timestamp" : 1367722894963,
> "child\_id" : 2030
> },
> "upsert" : {
> "children": \[
> {
> "id": 2030,
> "sent": 1367722894963
> }
> \]
> }
> }' && echo
> \`\`\`
> 
> Haven't taken a deeper look at it, but can confirm the curl call hangs and there is this exception in the logs:
> 
> \`\`\`
> \*\*\*\* COMPILER BUG! REPORT THIS IMMEDIATELY AT http://jira.codehaus.org/browse/mvel2
> Expression: if(ctx.\_source\["children"\] != null) { for (int i = 0; i \< ctx.\_source.children.size(); i++){ if(ctx.\_source.children\[i\].id == child\_id){ctx.\_source.children.remove(i);i--;}}} if (ctx.\_source\["children"\] == null) { ctx.\_source.children = \[{"id": child\_id, "sent": timestamp }\] } else {ctx.\_source.children += \[{"id": child\_id, "sent": timestamp }\] }
> Exception in thread "elasticsearch\[Acrobat\]\[index\]\[T#1\]" java.lang.VerifyError: (class: ASMAccessorImpl\_10448523421368185357780, method: getValue signature: (Ljava/lang/Object;Ljava/lang/Object;Lorg/elasticsearch/common/mvel2/integration/VariableResolverFactory;)Ljava/lang/Object;) Expecting to find integer on stack
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2398)
> at java.lang.Class.getConstructor0(Class.java:2708)
> at java.lang.Class.newInstance0(Class.java:328)
> at java.lang.Class.newInstance(Class.java:310)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.\_initializeAccessor(ASMAccessorOptimizer.java:725)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.compileAccessor(ASMAccessorOptimizer.java:859)
> at org.elasticsearch.common.mvel2.optimizers.impl.asm.ASMAccessorOptimizer.optimizeAccessor(ASMAccessorOptimizer.java:243)
> at org.elasticsearch.common.mvel2.optimizers.dynamic.DynamicGetAccessor.optimize(DynamicGetAccessor.java:90)
> at org.elasticsearch.common.mvel2.optimizers.dynamic.DynamicGetAccessor.getValue(DynamicGetAccessor.java:64)
> at org.elasticsearch.common.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108)
> at org.elasticsearch.common.mvel2.ast.BinaryOperation.getReducedValueAccelerated(BinaryOperation.java:108)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.IfNode.getReducedValueAccelerated(IfNode.java:73)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.ForNode.getReducedValueAccelerated(ForNode.java:67)
> at org.elasticsearch.common.mvel2.compiler.ExecutableAccessor.getValue(ExecutableAccessor.java:38)
> at org.elasticsearch.common.mvel2.ast.IfNode.getReducedValueAccelerated(IfNode.java:74)
> at org.elasticsearch.common.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
> at org.elasticsearch.common.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:113)
> at org.elasticsearch.script.mvel.MvelScriptEngineService$MvelExecutableScript.run(MvelScriptEngineService.java:130)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:310)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:211)
> at org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:85)
> at org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$AsyncSingleAction$1.run(TransportInstanceSingleOperationAction.java:191)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
> at java.lang.Thread.run(Thread.java:680)
> \`\`\`

--Alex

On Mon, May 13, 2013 at 1:26 AM, rockbobsta [bob@figjamit.com.au](mailto:bob@figjamit.com.au) wrote:

> Hi Alex,
> 
> no probs, I'll update if I find out any more.  
> thanks for the quick response!
> 
> regards  
> Bob.
> 
> On Friday, May 10, 2013 9:43:00 PM UTC+10, Alexander Reelsen wrote:
> 
> > Hey
> > 
> > thanks a lot for taking your time to create a reproducable report. I've  
> > created an issue at [https://github.com/\*\*elasticsearch/elasticsearch/](https://github.com/**elasticsearch/elasticsearch/)\*\*  
> > issues/3021 [https://github.com/elasticsearch/elasticsearch/issues/3021](https://github.com/elasticsearch/elasticsearch/issues/3021)  
> > Please add anything important there, if you find out more or I described  
> > something incorrectly. Thanks a lot!
> > 
> > --Alex
> > 
> > On Fri, May 10, 2013 at 3:53 AM, rockbobsta [b...@figjamit.com.au](mailto:b...@figjamit.com.au) wrote:
> > 
> > > Hi,
> > > 
> > > I'm having an intermittent problem with an upsert statement. The issue  
> > > appears to be in compiling the script, but only occurs under particular  
> > > circumstances.  
> > > I've seen two similar issues reported, but I believe this is slightly  
> > > different as it's using the standard script, not an MVEL script.  
> > > These reports mention a very similar stacktrace to what I'm seeing:  
> > > [https://groups.google.com/\*\*forum/?fromgroups=#!searchin/](https://groups.google.com/**forum/?fromgroups=#!searchin/)\*\*  
> > > elasticsearch/VerifyError/\*\*elasticsearch/aBnqKPyOl5E/**RyfJbVuCCcAJ[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ)  
> > > [https://groups.google.com/\*\*forum/?fromgroups=#!searchin/](https://groups.google.com/**forum/?fromgroups=#!searchin/)**  
> > > elasticsearch/VerifyError/\*\*elasticsearch/TpQL47BaG9k/\*\*8FFPPCIAOPwJ[https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ](https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ)
> > > 
> > > I've put together a quick gist to reproduce the problem:  
> > > [https://gist.github.com/\*\*baxford/5551877](https://gist.github.com/**baxford/5551877)[https://gist.github.com/baxford/5551877](https://gist.github.com/baxford/5551877)
> > > 
> > > Here are some more notes:
> > > 
> > > - I can reliably reproduce the problem if I create the index, insert the  
> > > 2 records in the gist, then stop elasticsearch, restart it, and try to  
> > > execute the upsert statement.
> > > - When the VerifyError occurs, the CURL request just hangs.
> > > - Interestingly, it appears to happen only on a document with a large  
> > > number of nested docs, while it doesn't appear to happen with a doc  
> > > containing a few nested docs
> > > - Even for the upsert that causes the VerifyError the first time, if I  
> > > cancel the CURL request then run it again, it works properly.
> > > 
> > > The main reason this is an issue is that we are running similar upsert  
> > > statements as this within and elasticsearch river. When the VerifyError  
> > > occurs, the whole thread seems to block indefinitely and the river stops  
> > > processing anything.
> > > 
> > > It's possible as suggested in other posts, that it's a bug in MVEL, but  
> > > it seems that i'm not using MVEL for my scripts (although possibly the  
> > > script I'm using gets compiled to MVEL at a later time?).  
> > > Here are some other details:  
> > > I first noticed this issue on a production AWS machine running:
> > > 
> > > - Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86\_64)
> > > - java version "1.7.0\_17"
> > > - Elasticsearch 0.20.6
> > > 
> > > To reproduce it, I'm running this locally on
> > > 
> > > - mac osx 10.7.5,
> > > - java version "1.6.0\_43"
> > > - latest master version of Elasticsearch from github (I'm using the  
> > > latest from github in case it's been fixed or is using a more recent  
> > > version of MVEL which fixes the problem)
> > > 
> > > Any assistance would be appreciated. I'll continue to debug and see if I  
> > > can get any more info on the cause or a fix.  
> > > thanks.
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com).
> > > 
> > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)  
> > > .
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:36am UTC](https://discuss.elastic.co/t/intermittent-verifyerror-in-update-upsert-script-with-nested-documents/11909/5 "2017-07-06T02:36:59Z")

</div>