Intermittent VerifyError in update/upsert script with nested documents

Elastic Stack Elasticsearch
1

Hi,

I'm having an intermittent problem with an upsert statement. The issue
appears to be in compiling the script, but only occurs under particular
circumstances.
I've seen two similar issues reported, but I believe this is slightly
different as it's using the standard script, not an MVEL script.
These reports mention a very similar stacktrace to what I'm seeing:
https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ
https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ

I've put together a quick gist to reproduce the problem:

Here are some more notes:

  • I can reliably reproduce the problem if I create the index, insert the 2
    records in the gist, then stop elasticsearch, restart it, and try to
    execute the upsert statement.
  • When the VerifyError occurs, the CURL request just hangs.
  • Interestingly, it appears to happen only on a document with a large
    number of nested docs, while it doesn't appear to happen with a doc
    containing a few nested docs
  • Even for the upsert that causes the VerifyError the first time, if I
    cancel the CURL request then run it again, it works properly.

The main reason this is an issue is that we are running similar upsert
statements as this within and elasticsearch river. When the VerifyError
occurs, the whole thread seems to block indefinitely and the river stops
processing anything.

It's possible as suggested in other posts, that it's a bug in MVEL, but it
seems that i'm not using MVEL for my scripts (although possibly the script
I'm using gets compiled to MVEL at a later time?).
Here are some other details:
I first noticed this issue on a production AWS machine running:

  • Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86_64)
  • java version "1.7.0_17"
  • Elasticsearch 0.20.6

To reproduce it, I'm running this locally on

  • mac osx 10.7.5,
  • java version "1.6.0_43"
  • latest master version of Elasticsearch from github (I'm using the latest
    from github in case it's been fixed or is using a more recent version of
    MVEL which fixes the problem)

Any assistance would be appreciated. I'll continue to debug and see if I
can get any more info on the cause or a fix.
thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey

thanks a lot for taking your time to create a reproducable report. I've
created an issue at


Please add anything important there, if you find out more or I described
something incorrectly. Thanks a lot!

--Alex

On Fri, May 10, 2013 at 3:53 AM, rockbobsta bob@figjamit.com.au wrote:

Hi,

I'm having an intermittent problem with an upsert statement. The issue
appears to be in compiling the script, but only occurs under particular
circumstances.
I've seen two similar issues reported, but I believe this is slightly
different as it's using the standard script, not an MVEL script.
These reports mention a very similar stacktrace to what I'm seeing:

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ

I've put together a quick gist to reproduce the problem:
https://gist.github.com/baxford/5551877

Here are some more notes:

  • I can reliably reproduce the problem if I create the index, insert the 2
    records in the gist, then stop elasticsearch, restart it, and try to
    execute the upsert statement.
  • When the VerifyError occurs, the CURL request just hangs.
  • Interestingly, it appears to happen only on a document with a large
    number of nested docs, while it doesn't appear to happen with a doc
    containing a few nested docs
  • Even for the upsert that causes the VerifyError the first time, if I
    cancel the CURL request then run it again, it works properly.

The main reason this is an issue is that we are running similar upsert
statements as this within and elasticsearch river. When the VerifyError
occurs, the whole thread seems to block indefinitely and the river stops
processing anything.

It's possible as suggested in other posts, that it's a bug in MVEL, but it
seems that i'm not using MVEL for my scripts (although possibly the script
I'm using gets compiled to MVEL at a later time?).
Here are some other details:
I first noticed this issue on a production AWS machine running:

  • Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86_64)
  • java version "1.7.0_17"
  • Elasticsearch 0.20.6

To reproduce it, I'm running this locally on

  • mac osx 10.7.5,
  • java version "1.6.0_43"
  • latest master version of Elasticsearch from github (I'm using the latest
    from github in case it's been fixed or is using a more recent version of
    MVEL which fixes the problem)

Any assistance would be appreciated. I'll continue to debug and see if I
can get any more info on the cause or a fix.
thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Alex,

no probs, I'll update if I find out any more.
thanks for the quick response!

regards
Bob.

On Friday, May 10, 2013 9:43:00 PM UTC+10, Alexander Reelsen wrote:

Hey

thanks a lot for taking your time to create a reproducable report. I've
created an issue at
https://github.com/elasticsearch/elasticsearch/issues/3021
Please add anything important there, if you find out more or I described
something incorrectly. Thanks a lot!

--Alex

On Fri, May 10, 2013 at 3:53 AM, rockbobsta <b...@figjamit.com.au<javascript:>

wrote:

Hi,

I'm having an intermittent problem with an upsert statement. The issue
appears to be in compiling the script, but only occurs under particular
circumstances.
I've seen two similar issues reported, but I believe this is slightly
different as it's using the standard script, not an MVEL script.
These reports mention a very similar stacktrace to what I'm seeing:

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ

https://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ

I've put together a quick gist to reproduce the problem:
https://gist.github.com/baxford/5551877

Here are some more notes:

  • I can reliably reproduce the problem if I create the index, insert the
    2 records in the gist, then stop elasticsearch, restart it, and try to
    execute the upsert statement.
  • When the VerifyError occurs, the CURL request just hangs.
  • Interestingly, it appears to happen only on a document with a large
    number of nested docs, while it doesn't appear to happen with a doc
    containing a few nested docs
  • Even for the upsert that causes the VerifyError the first time, if I
    cancel the CURL request then run it again, it works properly.

The main reason this is an issue is that we are running similar upsert
statements as this within and elasticsearch river. When the VerifyError
occurs, the whole thread seems to block indefinitely and the river stops
processing anything.

It's possible as suggested in other posts, that it's a bug in MVEL, but
it seems that i'm not using MVEL for my scripts (although possibly the
script I'm using gets compiled to MVEL at a later time?).
Here are some other details:
I first noticed this issue on a production AWS machine running:

  • Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86_64)
  • java version "1.7.0_17"
  • Elasticsearch 0.20.6

To reproduce it, I'm running this locally on

  • mac osx 10.7.5,
  • java version "1.6.0_43"
  • latest master version of Elasticsearch from github (I'm using the
    latest from github in case it's been fixed or is using a more recent
    version of MVEL which fixes the problem)

Any assistance would be appreciated. I'll continue to debug and see if I
can get any more info on the cause or a fix.
thanks.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hey,

Igor commented in the ticket meanwhile, maybe it helps you... at least to
switch from MVEL to javascript or similar :slight_smile:

--Alex

On Mon, May 13, 2013 at 1:26 AM, rockbobsta bob@figjamit.com.au wrote:

Hi Alex,

no probs, I'll update if I find out any more.
thanks for the quick response!

regards
Bob.

On Friday, May 10, 2013 9:43:00 PM UTC+10, Alexander Reelsen wrote:

Hey

thanks a lot for taking your time to create a reproducable report. I've
created an issue at https://github.com/elasticsearch/elasticsearch/
issues/3021 https://github.com/elasticsearch/elasticsearch/issues/3021
Please add anything important there, if you find out more or I described
something incorrectly. Thanks a lot!

--Alex

On Fri, May 10, 2013 at 3:53 AM, rockbobsta b...@figjamit.com.au wrote:

Hi,

I'm having an intermittent problem with an upsert statement. The issue
appears to be in compiling the script, but only occurs under particular
circumstances.
I've seen two similar issues reported, but I believe this is slightly
different as it's using the standard script, not an MVEL script.
These reports mention a very similar stacktrace to what I'm seeing:
https://groups.google.com/forum/?fromgroups=#!searchin/
elasticsearch/VerifyError/**elasticsearch/aBnqKPyOl5E/**RyfJbVuCCcAJhttps://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/aBnqKPyOl5E/RyfJbVuCCcAJ
https://groups.google.com/forum/?fromgroups=#!searchin/
elasticsearch/VerifyError/**elasticsearch/TpQL47BaG9k/**8FFPPCIAOPwJhttps://groups.google.com/forum/?fromgroups=#!searchin/elasticsearch/VerifyError/elasticsearch/TpQL47BaG9k/8FFPPCIAOPwJ

I've put together a quick gist to reproduce the problem:
https://gist.github.com/**baxford/5551877https://gist.github.com/baxford/5551877

Here are some more notes:

  • I can reliably reproduce the problem if I create the index, insert the
    2 records in the gist, then stop elasticsearch, restart it, and try to
    execute the upsert statement.
  • When the VerifyError occurs, the CURL request just hangs.
  • Interestingly, it appears to happen only on a document with a large
    number of nested docs, while it doesn't appear to happen with a doc
    containing a few nested docs
  • Even for the upsert that causes the VerifyError the first time, if I
    cancel the CURL request then run it again, it works properly.

The main reason this is an issue is that we are running similar upsert
statements as this within and elasticsearch river. When the VerifyError
occurs, the whole thread seems to block indefinitely and the river stops
processing anything.

It's possible as suggested in other posts, that it's a bug in MVEL, but
it seems that i'm not using MVEL for my scripts (although possibly the
script I'm using gets compiled to MVEL at a later time?).
Here are some other details:
I first noticed this issue on a production AWS machine running:

  • Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-36-virtual x86_64)
  • java version "1.7.0_17"
  • Elasticsearch 0.20.6

To reproduce it, I'm running this locally on

  • mac osx 10.7.5,
  • java version "1.6.0_43"
  • latest master version of Elasticsearch from github (I'm using the
    latest from github in case it's been fixed or is using a more recent
    version of MVEL which fixes the problem)

Any assistance would be appreciated. I'll continue to debug and see if I
can get any more info on the cause or a fix.
thanks.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5