# Incorrect query matches using NOT

**URL:** <https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915>\
**Category:** Elasticsearch\
**Created:** [December 2, 2012, 1:09pm UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915 "2012-12-02T13:09:18Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anton](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Anton](https://discuss.elastic.co/u/Anton)\
**Post date:** [December 2, 2012, 1:09pm UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915/1 "2012-12-02T13:09:18Z")

</div>

Hi all,

I have an indexing problem I can't seem to figure out. Say I'm looking for  
the keyword "UPS" in data from Twitter. I have many of them:

{  
"query" : { "query\_string" : {"query" : "interaction.content:'ups'"} },  
"size": 0  
}

_Returns: 133025 results found._

When I do the inverse, and look for content not mentioning the keyword  
"UPS", I get even more:

{  
"query" : { "query\_string" : {"query" : "NOT interaction.content:'ups'"} },  
"size": 0  
}

_Returns: 272485 results found._

However, when I take a closer look at the results supposedly not containing  
the keyword "UPS", I find:

["content"]=\> "Twitter meet ups are probably the gayest stupidest things  
ever"

["content"]=\> "aiiii beaaa dejameeee jopetaaaas!!!!! Ups ostia pa mi:o  
(cara whats app) JAJAJA tiamo"

["content"]=\> ""Im Fahrzeug für Zustellung" ist UPS's Art dir zu sagen: "du  
hast was vor? Vergiss es!""

["content"]=\> "Bon UPS tu te dépêches!!! Je dois sortir -\_-»"

["content"]=\> "Huh, iOS recognises UPS tracking numbers and gives an option  
to track on long press? That?s cool."

Clearly, these all contain the keyword "UPS", but for some reason haven't  
been indexed with that token. My first guess would be that my analyzers are  
causing this, but in that case the results correctly matching the keyword  
(in the same languages, using the same analyzers) would have been affected  
as well. I also have the keyword "UPS" in my list of protected words, so it  
should not be stemmed or changed during indexing.

Any ideas on what could be happening here?

Thanks,

Anton.

--

---

<div class="post-metadata">

**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\
**Post date:** [December 3, 2012, 3:41am UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915/2 "2012-12-03T03:41:53Z")

</div>

What's the curl command to create your index and the interaction.content  
mapping?

On Sunday, December 2, 2012 6:09:18 AM UTC-7, Anton wrote:

> Hi all,
> 
> I have an indexing problem I can't seem to figure out. Say I'm looking for  
> the keyword "UPS" in data from Twitter. I have many of them:
> 
> {  
> "query" : { "query\_string" : {"query" : "interaction.content:'ups'"} },  
> "size": 0  
> }
> 
> _Returns: 133025 results found._
> 
> When I do the inverse, and look for content not mentioning the keyword  
> "UPS", I get even more:
> 
> {  
> "query" : { "query\_string" : {"query" : "NOT interaction.content:'ups'"}  
> },  
> "size": 0  
> }
> 
> _Returns: 272485 results found._
> 
> However, when I take a closer look at the results supposedly not  
> containing the keyword "UPS", I find:
> 
> ["content"]=\> "Twitter meet ups are probably the gayest stupidest things  
> ever"
> 
> ["content"]=\> "aiiii beaaa dejameeee jopetaaaas!!!!! Ups ostia pa mi:o  
> (cara whats app) JAJAJA tiamo"
> 
> ["content"]=\> ""Im Fahrzeug für Zustellung" ist UPS's Art dir zu sagen:  
> "du hast was vor? Vergiss es!""
> 
> ["content"]=\> "Bon UPS tu te dépêches!!! Je dois sortir -\_-»"
> 
> ["content"]=\> "Huh, iOS recognises UPS tracking numbers and gives an  
> option to track on long press? That?s cool."
> 
> Clearly, these all contain the keyword "UPS", but for some reason haven't  
> been indexed with that token. My first guess would be that my analyzers are  
> causing this, but in that case the results correctly matching the keyword  
> (in the same languages, using the same analyzers) would have been affected  
> as well. I also have the keyword "UPS" in my list of protected words, so it  
> should not be stemmed or changed during indexing.
> 
> Any ideas on what could be happening here?
> 
> Thanks,
> 
> Anton.

--

---

<div class="post-metadata">

**Author:** ![Anton](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Anton](https://discuss.elastic.co/u/Anton)\
**Post date:** [December 4, 2012, 7:09am UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915/3 "2012-12-04T07:09:00Z")

</div>

This is the mapping for interaction.content:

curl -XPUT [http://localhost:9200/media/raw/\_mapping](http://localhost:9200/media/raw/_mapping) -d '  
{  
"raw": {  
"properties": {  
"interaction": {  
"dynamic": "true",  
"properties": {  
"content": {  
"type": "multi\_field",  
"index\_name": "customtag",  
"index": "analyzed",  
"index\_analyzer": "index\_analyzer",  
"search\_analyzer": "search\_analyzer",  
"store": "yes",  
"fields": {  
"content": {  
"type": "string",  
"index": "analyzed"  
},  
"original": {  
"type": "string",  
"index": "not\_analyzed"  
}  
}  
}  
}  
}  
}  
}  
}  
'

In addition to the index\_analyzer, I also specify an analyzer per language  
(using \_analyzer) when I add the content. For English this would be  
analyzer\_en.

curl -XPUT [http://localhost:9200/fedex\_socmedia/](http://localhost:9200/fedex_socmedia/) -d '  
{  
"settings" : {  
"index": {  
"analysis": {  
"analyzer": {  
"index\_analyzer": {  
"tokenizer": "standard",  
"filter": ["lowercase", "keyword\_marker", "standard",  
"asciifolding", "my\_delimiter"],  
"char\_filter": "html\_strip"  
},  
"search\_analyzer": {  
"tokenizer": "standard",  
"filter": ["lowercase", "keyword\_marker", "standard", "stop",  
"asciifolding", "snowball"]  
},  
"analyzer\_en": {  
"type": "snowball",  
"language": "English",  
"stopwords\_path": "/opt/elasticsearch/stopwords\_en.txt"  
}  
},  
"filter": {  
"my\_delimiter": {  
"type": "word\_delimiter",  
"generate\_word\_parts": true,  
"catenate\_words": true,  
"catenate\_numbers": true,  
"preserve\_original": true,  
"split\_on\_numerics": true,  
"stem\_english\_possessive": true  
},  
"keyword\_marker": {  
"type": "keyword\_marker",  
"keywords\_path": "/opt/elasticsearch/protectedwords.txt"  
}  
}  
}  
}  
}  
}  
'

I don't have "ups" in my list of stopwords, but I do have it in my list of  
protected words.

And this is an example of an added article:

{  
"interaction": {  
"content": "Twitter meet ups are probably the gayest stupidest things  
ever",  
},  
"\_analyzer": "analyzer\_en"  
}

Anton.

On Monday, December 3, 2012 4:41:53 AM UTC+1, smonasco wrote:

> What's the curl command to create your index and the interaction.content  
> mapping?
> 
> On Sunday, December 2, 2012 6:09:18 AM UTC-7, Anton wrote:
> 
> > Hi all,
> > 
> > I have an indexing problem I can't seem to figure out. Say I'm looking  
> > for the keyword "UPS" in data from Twitter. I have many of them:
> > 
> > {  
> > "query" : { "query\_string" : {"query" : "interaction.content:'ups'"} },  
> > "size": 0  
> > }
> > 
> > _Returns: 133025 results found._
> > 
> > When I do the inverse, and look for content not mentioning the keyword  
> > "UPS", I get even more:
> > 
> > {  
> > "query" : { "query\_string" : {"query" : "NOT interaction.content:'ups'"}  
> > },  
> > "size": 0  
> > }
> > 
> > _Returns: 272485 results found._
> > 
> > However, when I take a closer look at the results supposedly not  
> > containing the keyword "UPS", I find:
> > 
> > ["content"]=\> "Twitter meet ups are probably the gayest stupidest things  
> > ever"
> > 
> > ["content"]=\> "aiiii beaaa dejameeee jopetaaaas!!!!! Ups ostia pa mi:o  
> > (cara whats app) JAJAJA tiamo"
> > 
> > ["content"]=\> ""Im Fahrzeug für Zustellung" ist UPS's Art dir zu sagen:  
> > "du hast was vor? Vergiss es!""
> > 
> > ["content"]=\> "Bon UPS tu te dépêches!!! Je dois sortir -\_-»"
> > 
> > ["content"]=\> "Huh, iOS recognises UPS tracking numbers and gives an  
> > option to track on long press? That?s cool."
> > 
> > Clearly, these all contain the keyword "UPS", but for some reason haven't  
> > been indexed with that token. My first guess would be that my analyzers are  
> > causing this, but in that case the results correctly matching the keyword  
> > (in the same languages, using the same analyzers) would have been affected  
> > as well. I also have the keyword "UPS" in my list of protected words, so it  
> > should not be stemmed or changed during indexing.
> > 
> > Any ideas on what could be happening here?
> > 
> > Thanks,
> > 
> > Anton.

--

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [December 4, 2012, 5:49pm UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915/4 "2012-12-04T17:49:49Z")

</div>

Lucene, and therefore Elasticsearch, does not handle NOT queries when they  
are the only term:

[http://lucene.apache.org/core/4\_0\_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#NOT](http://lucene.apache.org/core/4_0_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.html#NOT)

Not sure if Elasticsearch does anything different to help support this use  
case. One workaround in Lucene is to explicitly ask for everything in a  
query with the NOT query:

_:_ AND NOT interaction.content:'ups'

Of course, this query is highly inefficient. It is easier to rethink the  
query and avoiding using only a single NOT.

Cheers,

Ivan

On Sun, Dec 2, 2012 at 5:09 AM, Anton [anton@365analytics.com](mailto:anton@365analytics.com) wrote:

> Hi all,
> 
> I have an indexing problem I can't seem to figure out. Say I'm looking for  
> the keyword "UPS" in data from Twitter. I have many of them:
> 
> {  
> "query" : { "query\_string" : {"query" : "interaction.content:'ups'"} },  
> "size": 0  
> }
> 
> _Returns: 133025 results found._
> 
> When I do the inverse, and look for content not mentioning the keyword  
> "UPS", I get even more:
> 
> {  
> "query" : { "query\_string" : {"query" : "NOT interaction.content:'ups'"}  
> },  
> "size": 0  
> }
> 
> _Returns: 272485 results found._
> 
> However, when I take a closer look at the results supposedly not  
> containing the keyword "UPS", I find:
> 
> ["content"]=\> "Twitter meet ups are probably the gayest stupidest things  
> ever"
> 
> ["content"]=\> "aiiii beaaa dejameeee jopetaaaas!!!!! Ups ostia pa mi:o  
> (cara whats app) JAJAJA tiamo"
> 
> ["content"]=\> ""Im Fahrzeug für Zustellung" ist UPS's Art dir zu sagen:  
> "du hast was vor? Vergiss es!""
> 
> ["content"]=\> "Bon UPS tu te dépêches!!! Je dois sortir -\_-»"
> 
> ["content"]=\> "Huh, iOS recognises UPS tracking numbers and gives an  
> option to track on long press? That?s cool."
> 
> Clearly, these all contain the keyword "UPS", but for some reason haven't  
> been indexed with that token. My first guess would be that my analyzers are  
> causing this, but in that case the results correctly matching the keyword  
> (in the same languages, using the same analyzers) would have been affected  
> as well. I also have the keyword "UPS" in my list of protected words, so it  
> should not be stemmed or changed during indexing.
> 
> Any ideas on what could be happening here?
> 
> Thanks,
> 
> Anton.
> 
> --

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/incorrect-query-matches-using-not/9915/5 "2017-07-06T03:01:31Z")

</div>

Closed on July 6, 2017, 3:01am UTC