Records neglected in search result


(Kartavya) #1

Hi

I am getting few record missing in search result even if the record like a
hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but at
the same time when I am searching for "abc AND def" then record appear in
the result.

I am wondering whether there is some criteria in ElasticSearch based on
score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--


(Lukáš Vlček) #2

Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal pulkitdotcom@gmail.comwrote:

Hi

I am getting few record missing in search result even if the record like a
hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but at
the same time when I am searching for "abc AND def" then record appear in
the result.

I am wondering whether there is some criteria in ElasticSearch based on
score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--


(Kartavya) #3

Sure Lukas

My first query:-
curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND
marriage))"}},{"field":{"UserGender":"F OR M OR
U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR
X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND
OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'
*
*
It return count for term "great" as 207 but When I run the same query with
term "great" in AND the count changes to 322 but it should be 207. Second
query is a below.

curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage)) AND
(great)
"}},{"field":{"UserGender":"F
OR M OR U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S
OR R OR X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR
HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

Now term "great" count changes to 302.
*
*
Please suggest something.
*
*
Thanks,
*
*

On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal pulkitdotcom@gmail.comwrote:

Hi

I am getting few record missing in search result even if the record like
a hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but at
the same time when I am searching for "abc AND def" then record appear in
the result.

I am wondering whether there is some criteria in ElasticSearch based on
score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--

--


(David Pilato) #4

Hi,

Here is an example of a full curl recreation. https://gist.github.com/2965777
Please provide something similar. It will be easier to replay your case.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:11, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Sure Lukas

My first query:-
curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d '{"query":{"filtered": {"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR (Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage))"}},{"field":{"UserGender":"F OR M OR U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } , "filter": {"range":{"CreatedAt": {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

It return count for term "great" as 207 but When I run the same query with term "great" in AND the count changes to 322 but it should be 207. Second query is a below.

curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d '{"query":{"filtered": {"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR (Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage)) AND (great)"}},{"field":{"UserGender":"F OR M OR U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } , "filter": {"range":{"CreatedAt": {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

Now term "great" count changes to 302.

Please suggest something.

Thanks,

On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:
Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal pulkitdotcom@gmail.com wrote:
Hi

I am getting few record missing in search result even if the record like a hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but at the same time when I am searching for "abc AND def" then record appear in the result.

I am wondering whether there is some criteria in ElasticSearch based on score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--

--


(Kartavya) #5

Hope, it is more readable now

My first query:-
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": "((gayANDMarriage) OR
(MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage))"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID":
"BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry":
"CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea":
"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}
*
*
It return count for term "great" as 207 but When I run the same query with
term "great" in AND the count changes to 322 but it should be 207. Second
query is a below.

{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": " ((gayANDMarriage) OR
(MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage)) AND (great)"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID":
"BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry":
"CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea":
"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}

Now term "great" count changes to 302.
*
*
Please suggest something.
*
*
Thanks,
*
*

On Mon, Sep 17, 2012 at 3:49 PM, David Pilato david@pilato.fr wrote:

Hi,

Here is an example of a full curl recreation.
https://gist.github.com/2965777
Please provide something similar. It will be easier to replay your case.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:11, Pulkit Agrawal pulkitdotcom@gmail.com a
écrit :

Sure Lukas

My first query:-
curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND
marriage))"}},{"field":{"UserGender":"F OR M OR
U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR
X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND
OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'
*
*
It return count for term "great" as 207 but When I run the same query
with term "great" in AND the count changes to 322 but it should be 207.
Second query is a below.

curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage)) *AND (great)
*"}},{"field":{"UserGender":"F OR M OR
U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR
X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND
OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

Now term "great" count changes to 302.
*
*
Please suggest something.
*
*
Thanks,
*
*

On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal pulkitdotcom@gmail.comwrote:

Hi

I am getting few record missing in search result even if the record like
a hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but
at the same time when I am searching for "abc AND def" then record appear
in the result.

I am wondering whether there is some criteria in ElasticSearch based on
score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--

--

--

--


(David Pilato) #6

I think you did not understand what Lukas and I are saying !

If I run your query on my ES instance. It won't work.
How can we help you if we are able to reproduce your use case?

We don't know your settings, your mappings, your docs...

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:30, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Hope, it is more readable now

My first query:-
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": "((gayANDMarriage) OR (MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage))"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID": "BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry": "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea": "CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}

It return count for term "great" as 207 but When I run the same query with term "great" in AND the count changes to 322 but it should be 207. Second query is a below.

{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": " ((gayANDMarriage) OR (MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage)) AND (great)"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID": "BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry": "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea": "CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}

Now term "great" count changes to 302.

Please suggest something.

Thanks,

On Mon, Sep 17, 2012 at 3:49 PM, David Pilato david@pilato.fr wrote:
Hi,

Here is an example of a full curl recreation. https://gist.github.com/2965777
Please provide something similar. It will be easier to replay your case.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:11, Pulkit Agrawal pulkitdotcom@gmail.com a écrit :

Sure Lukas

My first query:-
curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d '{"query":{"filtered": {"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR (Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage))"}},{"field":{"UserGender":"F OR M OR U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } , "filter": {"range":{"CreatedAt": {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

It return count for term "great" as 207 but When I run the same query with term "great" in AND the count changes to 322 but it should be 207. Second query is a below.

curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d '{"query":{"filtered": {"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR (Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR (Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR (Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR (Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one AND man AND one AND woman) OR (same AND gender AND marriage)) AND (great)"}},{"field":{"UserGender":"F OR M OR U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } , "filter": {"range":{"CreatedAt": {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

Now term "great" count changes to 302.

Please suggest something.

Thanks,

On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:
Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal pulkitdotcom@gmail.com wrote:
Hi

I am getting few record missing in search result even if the record like a hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but at the same time when I am searching for "abc AND def" then record appear in the result.

I am wondering whether there is some criteria in ElasticSearch based on score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--

--

--

--


(Kartavya) #7

here is my mapping of index

curl -XPUT 'localhost:9200/inn/innt/_mapping' -d '{
"innt": {
"properties": {
"FBLikeCount": {
"type": "integer"
},
"FBShareCount": {
"type": "integer"
},
"FBRecommendCount": {
"type": "integer"
},
"FBTotalCount": {
"type": "integer"
},
"FBClickCount": {
"type": "integer"
},
"TWShareCount": {
"type": "integer"
},
"NoOfComments": {
"type": "integer"
},
"VideoWatchCount": {
"type": "integer"
},
"AuthorNumberOfResult1": {
"type": "integer"
},
"polarity": {
"type": "multi_field",
"fields": {
"polarity": {
"type": "string"
},
"polarityint": {
"type": "integer"
}
}
}
}
}
}}'

Here is my sample document:

{
"_id": "245745919428014081",
"_rev": "1-e863bfb49a313b61f1a07cf3224dad49",
"SystemCreatedAt": "2012-09-12T05:13:32",
"MsgData": "I shoulda taken my medicine. Caught a bad case of feelings.",
"IsTweet": "Y",
"CreatedAt": "2012-9-12T04:49:27",
"Retweet": "N",
"SystemModifiedAt": "2012-09-12T05:13:32",
"ReplyTweetID": "",
"ReplyUserID": "",
"IsEnglish": "Y",
"polarity": "2",
"MessageEfluencerIndex": 91409.825,
"BusinessAreaConfiguration": "UD",
"Feature": "",
"LifeStyle": " HLT",
"Source": "Echofon",
"AuthorLocationKnow": "Y",
"AuthorLocationContinentCode": "NORTHAMERICA",
"AuthorLocationCountryCode": "US",
"AuthorLocationStateCode": " WY",
"GeoEnabled": "N",
"PlaceFullName": "",
"PlaceName": "",
"TweetContry": "",
"TweetCountryCode": "",
"TweetLatitude": "",
"TweetLongigttude": "",
"UserID": "23914462",
"UserName": "Proto_Zsay",
"ToUserID": "",
"ToUserName": "",
"IsPopular": "N",
"UserGender": "M",
"Name": "Pres. Chill Clinton",
"Location": "Byron Bay, NSW",
"UserEfluncerIndex": 18281.765,
"AuthorLocationRegionCode": "W",
"RetweetCount": "0",
"DataSourceTypeID": "T",
"DataSourceIndustry": "All",
"DataSourceBusinessArea": "All",
"DataSourceCoverageArea": "All"
}

Other than that there is no change in setting. I am using default
ElasticSearch setting of 0.18.7.

Let me know, if you are looking for any other information. I am happy to
provide the same.

Thanks,

On Mon, Sep 17, 2012 at 4:16 PM, David Pilato david@pilato.fr wrote:

I think you did not understand what Lukas and I are saying !

If I run your query on my ES instance. It won't work.
How can we help you if we are able to reproduce your use case?

We don't know your settings, your mappings, your docs...

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:30, Pulkit Agrawal pulkitdotcom@gmail.com a
écrit :

Hope, it is more readable now

My first query:-
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": "((gayANDMarriage) OR
(MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage))"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID":
"BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry":
"CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea":
"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}
*
*
It return count for term "great" as 207 but When I run the same query
with term "great" in AND the count changes to 322 but it should be 207.
Second query is a below.

{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"field": {
"MsgData": " ((gayANDMarriage) OR
(MarriageANDequality) OR (LGBTANDmarriage) OR (SamesexANDmarriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND ) OR (Marriage AND quality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage)) *AND (great)
*"
}
},
{
"field": {
"UserGender": "FORMORU"
}
},
{
"field": {
"DataSourceTypeID":
"BORTORNORDORFORVORSORRORX"
}
},
{
"field": {
"DataSourceIndustry":
"CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
}
},
{
"field": {
"DataSourceBusinessArea":
"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
}
},
{
"field": {
"polarity": "2OR1"
}
}
],
"minimum_number_should_match": 6
}
},
"filter": {
"range": {
"CreatedAt": {
"from": "2012-9-10T00: 00: 00",
"to": "2012-9-15T23: 59: 59"
}
}
}
}
},
"facets": {
"MsgData": {
"terms": {
"field": "MsgData",
"size": 100,
"exclude": [
"bit.ly",
"rt",
"http",
"t.co",
"i",
"you",
"my",
"your"
]
}
}
}
}

Now term "great" count changes to 302.
*
*
Please suggest something.
*
*
Thanks,
*
*

On Mon, Sep 17, 2012 at 3:49 PM, David Pilato david@pilato.fr wrote:

Hi,

Here is an example of a full curl recreation.
https://gist.github.com/2965777
Please provide something similar. It will be easier to replay your case.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 17 sept. 2012 à 12:11, Pulkit Agrawal pulkitdotcom@gmail.com a
écrit :

Sure Lukas

My first query:-
curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND
marriage))"}},{"field":{"UserGender":"F OR M OR
U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR
X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND
OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'
*
*
It return count for term "great" as 207 but When I run the same query
with term "great" in AND the count changes to 322 but it should be 207.
Second query is a below.

curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
'{"query":{"filtered":
{"query":{"bool":{"should":[{"field":{"MsgData":"((gay AND Marriage) OR
(Marriage AND equality) OR (LGBT AND marriage) OR (Samesex AND marriage) OR
(Prop8) OR (Samesex) OR (Marriageequality AND New AND Jersey) OR
(Marriageequality AND Minnesota) OR (Marriageequality AND New AND YORk) OR
(Marriageequality AND MarylAND) OR (Marriage AND Equality AND Act) OR
(Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt) OR (Molly
AND McKay) OR (Davina AND Kotulski) OR (Marriageequality AND ban) OR (one
AND man AND one AND woman) OR (same AND gender AND marriage)) AND
(great)
"}},{"field":{"UserGender":"F OR M OR
U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR F OR V OR S OR R OR
X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY OR FNS OR HLC OR IND
OR ITC OR MAT OR TEL OR UD OR UTL OR All OR
U"}},{"field":{"DataSourceBusinessArea":"CRS OR CUS OR MKI OR PRC OR PF OR
PTL OR CGN OR FNI OR LGI OR MA OR OPE OR PLT OR TNT OR UD OR All OR
U"}},{"field":{"polarity":"2 OR 1"}}],"minimum_number_should_match":6} } ,
"filter": {"range":{"CreatedAt":
{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

Now term "great" count changes to 302.
*
*
Please suggest something.
*
*
Thanks,
*
*

On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

I think you can get better help if you provide culr recreation.
http://www.elasticsearch.org/help/

Regards,
Lukas

On Sat, Sep 15, 2012 at 10:19 AM, Pulkit Agrawal <pulkitdotcom@gmail.com

wrote:

Hi

I am getting few record missing in search result even if the record
like a hit for query.

For Example:
Record in index is like- {"msg":"abc def"}
When I am searching for "abc" the record in not included in result but
at the same time when I am searching for "abc AND def" then record appear
in the result.

I am wondering whether there is some criteria in ElasticSearch based on
score or something else then ignore some records in result.

Please help.

Thanks,
Pulkit Agrawal

--

--

--

--

--

--

--


(Clinton Gormley) #8

Hi Pulkit

Have a look at http://www.elasticsearch.org/help

It explains how to provide code samples to make it easy for others to
debug problems. If we can't copy and paste something, then we are
unlikely to try it.

Other than that there is no change in setting. I am using default
ElasticSearch setting of 0.18.7.

Also, I would upgrade elasticsearch. A number of bugs have been fixed
since 0.18.7

clint

Let me know, if you are looking for any other information. I am happy
to provide the same.

Thanks,

On Mon, Sep 17, 2012 at 4:16 PM, David Pilato david@pilato.fr wrote:
I think you did not understand what Lukas and I are saying !

    If I run your query on my ES instance. It won't work.
    How can we help you if we are able to reproduce your use case?
    
    
    We don't know your settings, your mappings, your docs...
    
    --
    David ;-)
    Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
    
    
    
    Le 17 sept. 2012 à 12:30, Pulkit Agrawal
    <pulkitdotcom@gmail.com> a écrit :
    
    
    
    Hope, it is more readable now
    
    
    My first query:-
    {
        "query": {
            "filtered": {
                "query": {
                    "bool": {
                        "should": [
                            {
                                "field": {
                                    "MsgData": "((gayANDMarriage)
    OR (MarriageANDequality) OR (LGBTANDmarriage) OR
    (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR
    (Marriageequality AND New AND Jersey) OR (Marriageequality AND
    Minnesota) OR (Marriageequality AND New AND YORk) OR
    (Marriageequality AND MarylAND ) OR (Marriage AND quality AND
    Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND
    DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR
    (Marriageequality AND ban) OR (one AND man AND one AND woman)
    OR (same AND gender AND marriage))"
                                }
                            },
                            {
                                "field": {
                                    "UserGender": "FORMORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceTypeID":
    "BORTORNORDORFORVORSORRORX"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceIndustry":
    "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceBusinessArea":
    "CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "polarity": "2OR1"
                                }
                            }
                        ],
                        "minimum_number_should_match": 6
                    }
                },
                "filter": {
                    "range": {
                        "CreatedAt": {
                            "from": "2012-9-10T00: 00: 00",
                            "to": "2012-9-15T23: 59: 59"
                        }
                    }
                }
            }
        },
        "facets": {
            "MsgData": {
                "terms": {
                    "field": "MsgData",
                    "size": 100,
                    "exclude": [
                        "bit.ly",
                        "rt",
                        "http",
                        "t.co",
                        "i",
                        "you",
                        "my",
                        "your"
                    ]
                }
            }
        }
    }
    
    
    It return count for term "great" as 207 but When I run the
    same query with term "great" in AND the count changes to 322
    but it should be 207. Second query is a below.
    
    
    {
        "query": {
            "filtered": {
                "query": {
                    "bool": {
                        "should": [
                            {
                                "field": {
                                    "MsgData": " ((gayANDMarriage)
    OR (MarriageANDequality) OR (LGBTANDmarriage) OR
    (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR
    (Marriageequality AND New AND Jersey) OR (Marriageequality AND
    Minnesota) OR (Marriageequality AND New AND YORk) OR
    (Marriageequality AND MarylAND ) OR (Marriage AND quality AND
    Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND
    DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR
    (Marriageequality AND ban) OR (one AND man AND one AND woman)
    OR (same AND gender AND marriage)) AND (great)"
                                }
                            },
                            {
                                "field": {
                                    "UserGender": "FORMORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceTypeID":
    "BORTORNORDORFORVORSORRORX"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceIndustry":
    "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceBusinessArea":
    "CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "polarity": "2OR1"
                                }
                            }
                        ],
                        "minimum_number_should_match": 6
                    }
                },
                "filter": {
                    "range": {
                        "CreatedAt": {
                            "from": "2012-9-10T00: 00: 00",
                            "to": "2012-9-15T23: 59: 59"
                        }
                    }
                }
            }
        },
        "facets": {
            "MsgData": {
                "terms": {
                    "field": "MsgData",
                    "size": 100,
                    "exclude": [
                        "bit.ly",
                        "rt",
                        "http",
                        "t.co",
                        "i",
                        "you",
                        "my",
                        "your"
                    ]
                }
            }
        }
    }
    
    
    Now term "great" count changes to 302.
    
    
    Please suggest something.
    
    
    Thanks,
    
    
    
    On Mon, Sep 17, 2012 at 3:49 PM, David Pilato
    <david@pilato.fr> wrote:
            Hi,
            
            
            Here is an example of a full curl
            recreation. https://gist.github.com/2965777
            Please provide something similar. It will be easier to
            replay your case.
            
            --
            David ;-)
            Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
            
            
            
            Le 17 sept. 2012 à 12:11, Pulkit Agrawal
            <pulkitdotcom@gmail.com> a écrit :
            
            
            
            
            
            Sure Lukas
            
            
            My first query:-
            curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
            '{"query":{"filtered":
            {"query":{"bool":{"should":[{"field":{"MsgData":"((gay
            AND Marriage) OR (Marriage AND equality) OR (LGBT AND
            marriage) OR (Samesex AND marriage) OR (Prop8) OR
            (Samesex) OR (Marriageequality AND New AND Jersey) OR
            (Marriageequality AND Minnesota) OR (Marriageequality
            AND New AND YORk) OR (Marriageequality AND MarylAND)
            OR (Marriage AND Equality AND Act) OR (Connie AND
            Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt)
            OR (Molly AND McKay) OR (Davina AND Kotulski) OR
            (Marriageequality AND ban) OR (one AND man AND one AND
            woman) OR (same AND gender AND
            marriage))"}},{"field":{"UserGender":"F OR M OR
            U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR
            F OR V OR S OR R OR
            X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY
            OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL
            OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS
            OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI
            OR MA OR OPE OR PLT OR TNT OR UD OR All OR
            U"}},{"field":{"polarity":"2 OR
            1"}}],"minimum_number_should_match":6} } , "filter":
            {"range":{"CreatedAt":
            {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'
            
            
            It return count for term "great" as 207 but When I run
            the same query with term "great" in AND the count
            changes to 322 but it should be 207. Second query is a
            below.
            
            
            curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
            '{"query":{"filtered":
            {"query":{"bool":{"should":[{"field":{"MsgData":"((gay
            AND Marriage) OR (Marriage AND equality) OR (LGBT AND
            marriage) OR (Samesex AND marriage) OR (Prop8) OR
            (Samesex) OR (Marriageequality AND New AND Jersey) OR
            (Marriageequality AND Minnesota) OR (Marriageequality
            AND New AND YORk) OR (Marriageequality AND MarylAND)
            OR (Marriage AND Equality AND Act) OR (Connie AND
            Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt)
            OR (Molly AND McKay) OR (Davina AND Kotulski) OR
            (Marriageequality AND ban) OR (one AND man AND one AND
            woman) OR (same AND gender AND marriage)) AND
            (great)"}},{"field":{"UserGender":"F OR M OR
            U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR
            F OR V OR S OR R OR
            X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY
            OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL
            OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS
            OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI
            OR MA OR OPE OR PLT OR TNT OR UD OR All OR
            U"}},{"field":{"polarity":"2 OR
            1"}}],"minimum_number_should_match":6} } , "filter":
            {"range":{"CreatedAt":
            {"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} } },"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["bit.ly","rt","http","t.co","i","you","my","your""]}}}}'
            
            
            Now term "great" count changes to 302.
            
            
            Please suggest something.
            
            
            Thanks,
            
            
            
            On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček
            <lukas.vlcek@gmail.com> wrote:
                    Hi,
                    
                    
                    I think you can get better help if you provide
                    culr recreation.
                    http://www.elasticsearch.org/help/
                    
                    
                    Regards,
                    Lukas
                    
                    On Sat, Sep 15, 2012 at 10:19 AM, Pulkit
                    Agrawal <pulkitdotcom@gmail.com> wrote:
                    
                            Hi 
                            
                            
                            I am getting few record missing in
                            search result even if the record like
                            a hit for query.
                            
                            
                            For Example:
                            Record in index is like-  {"msg":"abc
                            def"}
                            When I am searching for "abc" the
                            record in not included in result but
                            at the same time when I am searching
                            for "abc AND def" then record appear
                            in the result.
                            
                            
                            I am wondering whether there is some
                            criteria in ElasticSearch based on
                            score or something else then ignore
                            some records in result.
                            
                            
                            Please help.
                            
                            
                            Thanks,
                            Pulkit Agrawal
                            -- 
                             
                             
                    
                    
                    
                    -- 
                     
                     
            
            -- 
             
             
            
            -- 
             
             
    
    -- 

--


(Kartavya) #9

Thanks.

I found same issue here

Now, can anybody help me?

I can not use single shard. What else I can do?

On Mon, Sep 17, 2012 at 4:33 PM, Clinton Gormley clint@traveljury.comwrote:

Hi Pulkit

Have a look at http://www.elasticsearch.org/help

It explains how to provide code samples to make it easy for others to
debug problems. If we can't copy and paste something, then we are
unlikely to try it.

Other than that there is no change in setting. I am using default
ElasticSearch setting of 0.18.7.

Also, I would upgrade elasticsearch. A number of bugs have been fixed
since 0.18.7

clint

Let me know, if you are looking for any other information. I am happy
to provide the same.

Thanks,

On Mon, Sep 17, 2012 at 4:16 PM, David Pilato david@pilato.fr wrote:
I think you did not understand what Lukas and I are saying !

    If I run your query on my ES instance. It won't work.
    How can we help you if we are able to reproduce your use case?


    We don't know your settings, your mappings, your docs...

    --
    David ;-)
    Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs



    Le 17 sept. 2012 à 12:30, Pulkit Agrawal
    <pulkitdotcom@gmail.com> a écrit :



    Hope, it is more readable now


    My first query:-
    {
        "query": {
            "filtered": {
                "query": {
                    "bool": {
                        "should": [
                            {
                                "field": {
                                    "MsgData": "((gayANDMarriage)
    OR (MarriageANDequality) OR (LGBTANDmarriage) OR
    (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR
    (Marriageequality AND New AND Jersey) OR (Marriageequality AND
    Minnesota) OR (Marriageequality AND New AND YORk) OR
    (Marriageequality AND MarylAND ) OR (Marriage AND quality AND
    Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND
    DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR
    (Marriageequality AND ban) OR (one AND man AND one AND woman)
    OR (same AND gender AND marriage))"
                                }
                            },
                            {
                                "field": {
                                    "UserGender": "FORMORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceTypeID":
    "BORTORNORDORFORVORSORRORX"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceIndustry":
    "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceBusinessArea":

"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"

                                }
                            },
                            {
                                "field": {
                                    "polarity": "2OR1"
                                }
                            }
                        ],
                        "minimum_number_should_match": 6
                    }
                },
                "filter": {
                    "range": {
                        "CreatedAt": {
                            "from": "2012-9-10T00: 00: 00",
                            "to": "2012-9-15T23: 59: 59"
                        }
                    }
                }
            }
        },
        "facets": {
            "MsgData": {
                "terms": {
                    "field": "MsgData",
                    "size": 100,
                    "exclude": [
                        "bit.ly",
                        "rt",
                        "http",
                        "t.co",
                        "i",
                        "you",
                        "my",
                        "your"
                    ]
                }
            }
        }
    }


    It return count for term "great" as 207 but When I run the
    same query with term "great" in AND the count changes to 322
    but it should be 207. Second query is a below.


    {
        "query": {
            "filtered": {
                "query": {
                    "bool": {
                        "should": [
                            {
                                "field": {
                                    "MsgData": " ((gayANDMarriage)
    OR (MarriageANDequality) OR (LGBTANDmarriage) OR
    (SamesexANDmarriage) OR (Prop8) OR (Samesex) OR
    (Marriageequality AND New AND Jersey) OR (Marriageequality AND
    Minnesota) OR (Marriageequality AND New AND YORk) OR
    (Marriageequality AND MarylAND ) OR (Marriage AND quality AND
    Act) OR (Connie AND Ress) OR (Michael AND Thurber) OR (Jim AND
    DeLaHunt) OR (Molly AND McKay) OR (Davina AND Kotulski) OR
    (Marriageequality AND ban) OR (one AND man AND one AND woman)
    OR (same AND gender AND marriage)) AND (great)"
                                }
                            },
                            {
                                "field": {
                                    "UserGender": "FORMORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceTypeID":
    "BORTORNORDORFORVORSORRORX"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceIndustry":
    "CDCORCSPOREGYORFNSORHLCORINDORITCORMATORTELORUDORUTLORAllORU"
                                }
                            },
                            {
                                "field": {
                                    "DataSourceBusinessArea":

"CRSORCUSORMKIORPRCORPFORPTLORCGNORFNIORLGIORMAOROPEORPLTORTNTORUDORAllORU"

                                }
                            },
                            {
                                "field": {
                                    "polarity": "2OR1"
                                }
                            }
                        ],
                        "minimum_number_should_match": 6
                    }
                },
                "filter": {
                    "range": {
                        "CreatedAt": {
                            "from": "2012-9-10T00: 00: 00",
                            "to": "2012-9-15T23: 59: 59"
                        }
                    }
                }
            }
        },
        "facets": {
            "MsgData": {
                "terms": {
                    "field": "MsgData",
                    "size": 100,
                    "exclude": [
                        "bit.ly",
                        "rt",
                        "http",
                        "t.co",
                        "i",
                        "you",
                        "my",
                        "your"
                    ]
                }
            }
        }
    }


    Now term "great" count changes to 302.


    Please suggest something.


    Thanks,



    On Mon, Sep 17, 2012 at 3:49 PM, David Pilato
    <david@pilato.fr> wrote:
            Hi,


            Here is an example of a full curl
            recreation. https://gist.github.com/2965777
            Please provide something similar. It will be easier to
            replay your case.

            --
            David ;-)
            Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs



            Le 17 sept. 2012 à 12:11, Pulkit Agrawal
            <pulkitdotcom@gmail.com> a écrit :





            Sure Lukas


            My first query:-
            curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
            '{"query":{"filtered":
            {"query":{"bool":{"should":[{"field":{"MsgData":"((gay
            AND Marriage) OR (Marriage AND equality) OR (LGBT AND
            marriage) OR (Samesex AND marriage) OR (Prop8) OR
            (Samesex) OR (Marriageequality AND New AND Jersey) OR
            (Marriageequality AND Minnesota) OR (Marriageequality
            AND New AND YORk) OR (Marriageequality AND MarylAND)
            OR (Marriage AND Equality AND Act) OR (Connie AND
            Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt)
            OR (Molly AND McKay) OR (Davina AND Kotulski) OR
            (Marriageequality AND ban) OR (one AND man AND one AND
            woman) OR (same AND gender AND
            marriage))"}},{"field":{"UserGender":"F OR M OR
            U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR
            F OR V OR S OR R OR
            X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY
            OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL
            OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS
            OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI
            OR MA OR OPE OR PLT OR TNT OR UD OR All OR
            U"}},{"field":{"polarity":"2 OR
            1"}}],"minimum_number_should_match":6} } , "filter":
            {"range":{"CreatedAt":

{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

            It return count for term "great" as 207 but When I run
            the same query with term "great" in AND the count
            changes to 322 but it should be 207. Second query is a
            below.


            curl -XGET 'localhost:9200XXX/YYYt/_search?size=0' -d
            '{"query":{"filtered":
            {"query":{"bool":{"should":[{"field":{"MsgData":"((gay
            AND Marriage) OR (Marriage AND equality) OR (LGBT AND
            marriage) OR (Samesex AND marriage) OR (Prop8) OR
            (Samesex) OR (Marriageequality AND New AND Jersey) OR
            (Marriageequality AND Minnesota) OR (Marriageequality
            AND New AND YORk) OR (Marriageequality AND MarylAND)
            OR (Marriage AND Equality AND Act) OR (Connie AND
            Ress) OR (Michael AND Thurber) OR (Jim AND DeLaHunt)
            OR (Molly AND McKay) OR (Davina AND Kotulski) OR
            (Marriageequality AND ban) OR (one AND man AND one AND
            woman) OR (same AND gender AND marriage)) AND
            (great)"}},{"field":{"UserGender":"F OR M OR
            U"}},{"field":{"DataSourceTypeID":"B OR T OR N OR D OR
            F OR V OR S OR R OR
            X"}},{"field":{"DataSourceIndustry":"CDC OR CSP OR EGY
            OR FNS OR HLC OR IND OR ITC OR MAT OR TEL OR UD OR UTL
            OR All OR U"}},{"field":{"DataSourceBusinessArea":"CRS
            OR CUS OR MKI OR PRC OR PF OR PTL OR CGN OR FNI OR LGI
            OR MA OR OPE OR PLT OR TNT OR UD OR All OR
            U"}},{"field":{"polarity":"2 OR
            1"}}],"minimum_number_should_match":6} } , "filter":
            {"range":{"CreatedAt":

{"from":"2012-9-10T00:00:00","to":"2012-9-15T23:59:59"}}} }
},"facets":{"MsgData":{"terms":{"field":"MsgData","size":100,"exclude":["
bit.ly","rt","http","t.co","i","you","my","your""]}}}}'

            Now term "great" count changes to 302.


            Please suggest something.


            Thanks,



            On Sun, Sep 16, 2012 at 12:13 AM, Lukáš Vlček
            <lukas.vlcek@gmail.com> wrote:
                    Hi,


                    I think you can get better help if you provide
                    culr recreation.
                    http://www.elasticsearch.org/help/


                    Regards,
                    Lukas

                    On Sat, Sep 15, 2012 at 10:19 AM, Pulkit
                    Agrawal <pulkitdotcom@gmail.com> wrote:

                            Hi


                            I am getting few record missing in
                            search result even if the record like
                            a hit for query.


                            For Example:
                            Record in index is like-  {"msg":"abc
                            def"}
                            When I am searching for "abc" the
                            record in not included in result but
                            at the same time when I am searching
                            for "abc AND def" then record appear
                            in the result.


                            I am wondering whether there is some
                            criteria in ElasticSearch based on
                            score or something else then ignore
                            some records in result.


                            Please help.


                            Thanks,
                            Pulkit Agrawal
                            --





                    --



            --



            --



    --

--

--


(system) #10