Major Bug in 0.17.x and 0.18.x (up to 0.18.2)


(Shay Banon) #1

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug means
that under load, with long running search requests, the node can become
unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon


(Patrick) #2

I had some reports from a few friends that when they tried to bump
their installations of 0.17 to 0.18 (for daikon), logstash didn't play
well with it. I'm not sure how much of a factor that would play, but
I'd assume that quite a few of those unfortunate souls would love the
upgrade, but worry about the breakage.

Patrick


patrick eefy net

On Tue, Nov 15, 2011 at 12:56 AM, Shay Banon kimchy@gmail.com wrote:

Hey,
I posted before as a reply to a thread, but it should be on its own
thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including
0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455. This
bug means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.
There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade to
0.18 while they are at it, but understand if there is a need for 0.17
release.
-shay.banon


(David Pilato) #3

At first, I would use 0.17 in production in the next weeks but I will switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kimchy@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455. This bug means that under load, with long running search requests, the node can become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be released either today or tomorrow. Wanted to get feedback if a 0.17.10 with the fix should be released as well. I prefer users who upgrade to upgrade to 0.18 while they are at it, but understand if there is a need for 0.17 release.

-shay.banon


(Shay Banon) #4

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato david@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kimchy@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon


(Lukáš Vlček) #5

Hi,

what is the chance of hitting this issue when requests are executed with a
timeout option with 0.18.2 ?

Regards,
Lukáš
Dne 15.11.2011 7:57 "Shay Banon" kimchy@gmail.com napsal(a):

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon


(Shay Banon) #6

Similar, though timeout will bound the execution (though its not perfect).

On Tue, Nov 15, 2011 at 10:18 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

what is the chance of hitting this issue when requests are executed with a
timeout option with 0.18.2 ?

Regards,
Lukáš
Dne 15.11.2011 7:57 "Shay Banon" kimchy@gmail.com napsal(a):

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2):
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon


(Shay Banon) #7

Can they provide more details? Maybe its because logstash embeds
elasticsearch, and upgrading the "server" side of elasticsearch to 0.18
also requires upgrading the embedded version in logstash as well.

On Tue, Nov 15, 2011 at 9:01 AM, Patrick patrick@eefy.net wrote:

I had some reports from a few friends that when they tried to bump
their installations of 0.17 to 0.18 (for daikon), logstash didn't play
well with it. I'm not sure how much of a factor that would play, but
I'd assume that quite a few of those unfortunate souls would love the
upgrade, but worry about the breakage.

Patrick

http://about.me/patrick.ancillotti
patrick eefy net

On Tue, Nov 15, 2011 at 12:56 AM, Shay Banon kimchy@gmail.com wrote:

Hey,
I posted before as a reply to a thread, but it should be on its own
thread and notice. A major bug was found in 0.17.x versions and 0.18.x
(up
to and including
0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455.
This
bug means that under load, with long running search requests, the node
can
become unresponsive and not handle requests properly.
There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10
with
the fix should be released as well. I prefer users who upgrade to
upgrade to
0.18 while they are at it, but understand if there is a need for 0.17
release.
-shay.banon


(medcl.net) #8

Hi,

I  just found an interesting issue,when setting filter‘s cache to be true, different position of the parameter “size”  will introduce different result size.

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from": 0,"size": 6,"explain": false}

it return 10 hits ,so there is not correct.

2.and now remove the cache parameter
{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} }},"boost": 1}},"from": 0,"size": 6,"explain": false}

what I got is 6 hits,k,it’s right.

3.now back to the first query,and do little change,I moved the “from” “size” in front of “query”
{"from": 0,"size": 6,"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"explain": false}

and now what I got is 6 hits.

I think this issue matters,so reported there. .
BTW,and a parameter to the url,such as“http://host:9200/index/type/_search?size=6” does works.

these tests and under clean environment.

yours sincerely,
medcl
http://log.medcl.net


(Shay Banon) #9

Its because the placement of the _cache element is incorrect, it should on
the same level as filter, not within it. I still need to check why it does
not fail properly in this case.

Btw, why such complex query? Seems like what you need is just a term filter
within the constant_score, why wrap it in a query filter that wraps a term
query?

On Tue, Nov 15, 2011 at 12:19 PM, medcl2000@gmail.com wrote:

Hi,

I  just found an interesting issue,when setting filter‘s cache to be

true, different position of the parameter “size” will introduce different
result size.

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term"
: "true", "boost":1 }} },"_cache": true},"boost": 1}},"from": 0*,"size": 6
*,"explain": false}

it return 10 hits ,so there is not correct.

2.and now remove the cache parameter
{"query":{"constant_score":{"filter":{"query":{term: { "gender" : { "term"
: "true", "boost":1 }} }},"boost": 1}},"from": 0,"size": 6,"explain":
false}

what I got is 6 hits,k,it’s right.

3.now back to the first query,and do little change,I moved the “from”
“size” in front of “query”
{"from": 0*,"size": 6*,"query":{"constant_score":{"filter":{"query":{term:
{ "gender" : { "term" : "true", "boost":1 }} },"_cache": true},"boost":
1}},"explain": false}

and now what I got is 6 hits.

I think this issue matters,so reported there. [image: Smile].
BTW,and a parameter to the url,such as“
http://host:9200/index/type/_search?size=6” does works.

these tests and under clean environment.

yours sincerely,
medcl
http://log.medcl.net


(Clinton Gormley) #10

Hi medcl

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" :
{ "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from":
0,"size": 6,"explain": false}

With a cached query filter, you need to wrap the "query" and "_cache"
parameters in an 'fquery' element:

"filter" : {
    fquery : {
        "query" : {
            term : {
                "gender" : { "term" : "true", "boost" : 1 }
            }
        },
        "_cache" : true
    }
},

http://www.elasticsearch.org/guide/reference/query-dsl/query-filter.html

That said, why are you using a query filter at all?

Your query would be better written as:

{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"gender" : "true"
}
}
}
}
}

And that filter would be cached by default.

clint


(medcl.net) #11

hi,
thanks for you quickly reply,I just tried to wrapper a queryDSL layer
,my bad,and seems lots of work need to do.

-----Original Message-----
From: Clinton Gormley
Sent: Tuesday, November 15, 2011 7:00 PM
To: elasticsearch@googlegroups.com
Subject: Re: issue about size when set filter cached

Hi medcl

1.set cache to true,and the “from” “size” are after “query”.

{"query":{"constant_score":{"filter":{"query":{term: { "gender" :
{ "term" : "true", "boost":1 }} },"_cache": true},"boost": 1}},"from":
0,"size": 6,"explain": false}

With a cached query filter, you need to wrap the "query" and "_cache"
parameters in an 'fquery' element:

"filter" : {
    fquery : {
        "query" : {
            term : {
                "gender" : { "term" : "true", "boost" : 1 }
            }
        },
        "_cache" : true
    }
},

http://www.elasticsearch.org/guide/reference/query-dsl/query-filter.html

That said, why are you using a query filter at all?

Your query would be better written as:

{
"query" : {
"constant_score" : {
"filter" : {
"term" : {
"gender" : "true"
}
}
}
}
}

And that filter would be cached by default.

clint


(Frederic) #12

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x (up
to and including 0.18.2): https://github.com/elasticsearch/elasticsearch/issues/1455
https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10 with
the fix should be released as well. I prefer users who upgrade to upgrade
to 0.18 while they are at it, but understand if there is a need for 0.17
release.

-shay.banon


(Shay Banon) #13

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo.br@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its own

thread and notice. A major bug was found in 0.17.x versions and 0.18.x
(up

to and including 0.18.2): <
https://github.com/elasticsearch/elasticsearch/issues/1455>

https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will be
released either today or tomorrow. Wanted to get feedback if a 0.17.10
with

the fix should be released as well. I prefer users who upgrade to
upgrade

to 0.18 while they are at it, but understand if there is a need for
0.17

release.

-shay.banon


(Shay Banon) #14

0.17.10 has just been released. I want to fix another bug in 0.18 and then
release 0.18.3 today.

On Tue, Nov 15, 2011 at 6:13 PM, Shay Banon kimchy@gmail.com wrote:

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo.br@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its

own

thread and notice. A major bug was found in 0.17.x versions and
0.18.x (up

to and including 0.18.2): <
https://github.com/elasticsearch/elasticsearch/issues/1455>

https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will
be

released either today or tomorrow. Wanted to get feedback if a
0.17.10 with

the fix should be released as well. I prefer users who upgrade to
upgrade

to 0.18 while they are at it, but understand if there is a need for
0.17

release.

-shay.banon


(Frederic) #15

That's great. Thanks a lot Shay

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and then
release 0.18.3 today.

On Tue, Nov 15, 2011 at 6:13 PM, Shay Banon kim...@gmail.com wrote:

One user is enough :), I will release 0.17.10 either tonight or tomorrow.

On Tue, Nov 15, 2011 at 5:02 PM, Frederic focampo...@gmail.com wrote:

Hi Shay,

Thanks for the update and fix. We would certainly need the 0.17.x fix
for our systems in production, while I test the upgrade to 0.18 in the
meantime.

Simple question though: in these case, is it equally required to
update clients as well? we have both Transport and Node clients
connected to our ES cluster.

Thanks,

On 15 nov, 04:42, Shay Banon kim...@gmail.com wrote:

I applied the fix to the 0.17 branch in any case, btw.

On Tue, Nov 15, 2011 at 9:06 AM, David Pilato da...@pilato.fr wrote:

At first, I would use 0.17 in production in the next weeks but I will
switch to 0.18 as you suggest.

So at the present time, I don't need a new release of 0.17.

Thanks
David :wink:
@dadoonet

Le 15 nov. 2011 à 07:56, Shay Banon kim...@gmail.com a écrit :

Hey,

I posted before as a reply to a thread, but it should be on its

own

thread and notice. A major bug was found in 0.17.x versions and
0.18.x (up

to and including 0.18.2): <
https://github.com/elasticsearch/elasticsearch/issues/1455>

https://github.com/elasticsearch/elasticsearch/issues/1455. This bug
means that under load, with long running search requests, the node can
become unresponsive and not handle requests properly.

There is already a fix in master and 0.18 branch, and 0.18.3 will
be

released either today or tomorrow. Wanted to get feedback if a
0.17.10 with

the fix should be released as well. I prefer users who upgrade to
upgrade

to 0.18 while they are at it, but understand if there is a need for
0.17

release.

-shay.banon


(Damien Hardy) #16

Hello,

Any chance having the .deb package directly available for download ?

Thank you for the good working,

--
Damien

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and
then
release 0.18.3 today.


(Shay Banon) #17

You mean the one that is built as part of elasticsearch since 0.18.x? It
will be provided as an official download once people will start to use it a
bit.

On Wed, Nov 16, 2011 at 4:08 PM, Damien Hardy damienhardy.bal@gmail.comwrote:

Hello,

Any chance having the .deb package directly available for download ?

Thank you for the good working,

--
Damien

On 16 nov, 07:54, Shay Banon kim...@gmail.com wrote:

0.17.10 has just been released. I want to fix another bug in 0.18 and
then
release 0.18.3 today.


(system) #18