Major Bug in 0.17.x and 0.18.x (up to 0.18.2)

Elastic Stack Elasticsearch
1

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug means
that under load, with long running search requests, the node can become
unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon

2

I had some reports from a few friends that when they tried to bump
their installations of 0.17 to 0.18 (for daikon), logstash didn't play
well with it. I'm not sure how much of a factor that would play, but
I'd assume that quite a few of those unfortunate souls would love the
upgrade, but worry about the breakage.

Patrick

patrick eefy net

On Tue, Nov 15, 2011 at 12:56 AM, Shay Banon kimchy@gmail.com wrote:

Hey,
I posted before as a reply to a thread, but it should be on its own
thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including
0.18.2): Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This
bug means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.
There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade to
0.18 while they are at it, but understand if there is a need for 0.17
release.
-shay.banon

3

At first, I would use 0.17 in production in the next weeks but I will switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kimchy@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455. This bug means that under load, with long running search requests, the node can become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be released either today or tomorrow. Wanted to get feedback if a 0.17.10 with the fix should be released as well. I prefer users who upgrade to upgrade to 0.18 while they are at it, but understand if there is a need for 0.17 release.

-shay.banon

4

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato david@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kimchy@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon

5

Hi,

what is the chance of hitting this issue when requests are executed with a
timeout option with 0.18.2 ?

Regards,
Lukáš
Dne 15.11.2011 7:57 "Shay Banon" kimchy@gmail.com napsal(a):

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon

6

Similar, though timeout will bound the execution (though its not perfect).

On Tue, Nov 15, 2011 at 10:18 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

what is the chance of hitting this issue when requests are executed with a
timeout option with 0.18.2 ?

Regards,
Lukáš
Dne 15.11.2011 7:57 "Shay Banon" kimchy@gmail.com napsal(a):

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon

7

Can they provide more details? Maybe its because logstash embeds
elasticsearch, and upgrading the "server" side of elasticsearch to 0.18
also requires upgrading the embedded version in logstash as well.

On Tue, Nov 15, 2011 at 9:01 AM, Patrick patrick@eefy.net wrote:

I had some reports from a few friends that when they tried to bump
their installations of 0.17 to 0.18 (for daikon), logstash didn't play
well with it. I'm not sure how much of a factor that would play, but
I'd assume that quite a few of those unfortunate souls would love the
upgrade, but worry about the breakage.

Patrick

Patrick Ancillotti - New York | about.me
patrick eefy net

On Tue, Nov 15, 2011 at 12:56 AM, Shay Banon kimchy@gmail.com wrote:

Hey,
I posted before as a reply to a thread, but it should be on its own
thread and notice. A major bug was found in 0.17.x versions and 0.18.x
(up
to and including
0.18.2): Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub.
This
bug means that under load, with long running search requests, the node
can
become unresponsive and not handle requests properly.
There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10
with
the fix should be released as well. I prefer users who upgrade to
upgrade to
0.18 while they are at it, but understand if there is a need for 0.17
release.
-shay.banon

8

Hi,

I  just found an interesting issue，when setting filter‘s cache to be true, different position of the parameter “size”  will introduce different result size.

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from": 0,"size": 6,"explain": false}

it return 10 hits ,so there is not correct.

2.and now remove the cache parameter
{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} }},"boost": 1}},"from": 0,"size": 6,"explain": false}

what I got is 6 hits,k,it’s right.

3.now back to the first query,and do little change,I moved the “from” “size” in front of “query”
{"from": 0,"size": 6,"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"explain": false}

and now what I got is 6 hits.

I think this issue matters,so reported there. .
BTW,and a parameter to the url,such as“http://host:9200/index/type/_search?size=6” does works.

these tests and under clean environment.

yours sincerely,
medcl
http://log.medcl.net

9

Its because the placement of the _cache element is incorrect, it should on
the same level as filter, not within it. I still need to check why it does
not fail properly in this case.

Btw, why such complex query? Seems like what you need is just a term filter
within the constant_score, why wrap it in a query filter that wraps a term
query?

On Tue, Nov 15, 2011 at 12:19 PM, medcl2000@gmail.com wrote:

Hi,

I  just found an interesting issue，when setting filter‘s cache to be

true, different position of the parameter “size” will introduce different
result size.

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term"
: "true", "boost":1 }} },"_cache": true},"boost": 1}},"from": 0*,"size": 6
*,"explain": false}

it return 10 hits ,so there is not correct.

2.and now remove the cache parameter
{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term"
: "true", "boost":1 }} }},"boost": 1}},"from": 0,"size": 6,"explain":
false}

what I got is 6 hits,k,it’s right.

3.now back to the first query,and do little change,I moved the “from”
“size” in front of “query”
{"from": 0*,"size": 6*,"query":{"constant_score":{"filter":{"query":{term:
{ "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost":
1}},"explain": false}

and now what I got is 6 hits.

I think this issue matters,so reported there. [image: Smile].
BTW,and a parameter to the url,such as“
http://host:9200/index/type/_search?size=6” does works.

these tests and under clean environment.

yours sincerely,
medcl
http://log.medcl.net

10

Hi medcl

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" :
{ "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from":
0,"size": 6,"explain": false}

With a cached query filter, you need to wrap the "query" and "_cache"
parameters in an 'fquery' element:

"filter" : {
    fquery : {
        "query" : {
            term : {
                "gender" : { "term" : "true", "boost" : 1 }
            }
        },
        "_cache" : true
    }
},

That said, why are you using a query filter at all?

Your query would be better written as:

{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"gender" : "true"
}
}
}
}
}

And that filter would be cached by default.

clint

11

hi,
thanks for you quickly reply,I just tried to wrapper a queryDSL layer
,my bad,and seems lots of work need to do.

-----Original Message-----
From: Clinton Gormley
Sent: Tuesday, November 15, 2011 7:00 PM
To: elasticsearch@googlegroups.com
Subject: Re: issue about size when set filter cached

Hi medcl

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" :
{ "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from":
0,"size": 6,"explain": false}

With a cached query filter, you need to wrap the "query" and "_cache"
parameters in an 'fquery' element:

"filter" : {
    fquery : {
        "query" : {
            term : {
                "gender" : { "term" : "true", "boost" : 1 }
            }
        },
        "_cache" : true
    }
},

That said, why are you using a query filter at all?

Your query would be better written as:

{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"gender" : "true"
}
}
}
}
}

And that filter would be cached by default.

clint

12

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon

13

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo.br@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x
(up
to and including 0.18.2): <
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub>
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10
with
the fix should be released as well. I prefer users who upgrade to
upgrade
to 0.18 while they are at it, but understand if there is a need for
0.17
release.

-shay.banon

14

0.17.10 has just been released. I want to fix another bug in 0.18 and then
release 0.18.3 today.

On Tue, Nov 15, 2011 at 6:13 PM, Shay Banon kimchy@gmail.com wrote:

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo.br@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its

own

thread and notice. A major bug was found in 0.17.x versions and
0.18.x (up
to and including 0.18.2): <
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub>
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will
be
released either today or tomorrow. Wanted to get feedback if a
0.17.10 with
the fix should be released as well. I prefer users who upgrade to
upgrade
to 0.18 while they are at it, but understand if there is a need for
0.17
release.

-shay.banon

15

That's great. Thanks a lot Shay

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and then
release 0.18.3 today.

On Tue, Nov 15, 2011 at 6:13 PM, Shay Banon kim...@gmail.com wrote:

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo...@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its

own

thread and notice. A major bug was found in 0.17.x versions and
0.18.x (up
to and including 0.18.2): <
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub>
Search: Search requests execute by mistake on the networking http IO thread, causing other http operations to hang · Issue #1455 · elastic/elasticsearch · GitHub. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will
be
released either today or tomorrow. Wanted to get feedback if a
0.17.10 with
the fix should be released as well. I prefer users who upgrade to
upgrade
to 0.18 while they are at it, but understand if there is a need for
0.17
release.

-shay.banon

16

Hello,

Any chance having the .deb package directly available for download ?

Thank you for the good working,

--
Damien

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and
then
release 0.18.3 today.

17

You mean the one that is built as part of elasticsearch since 0.18.x? It
will be provided as an official download once people will start to use it a
bit.

On Wed, Nov 16, 2011 at 4:08 PM, Damien Hardy damienhardy.bal@gmail.comwrote:

Hello,

Any chance having the .deb package directly available for download ?

Thank you for the good working,

--
Damien

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and
then
release 0.18.3 today.