Exclude some documents (and category filter combination) for some queries

Elastic Stack Elasticsearch
1

I would like to exclude some documents belonging to certain category from
the results only for certain search queries. I have a ES client layer where
i am thinking of implementing this logic as a "not" filter depending on the
search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201", category:
2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203", category:
5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for other
keywords) but filter out few categories from the result. As ES returns the
whole nested document even if only one nested document matches the query,
my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201", category:
2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203", category:
5} ]
Here is the query i tried but it doesn't work. Can anyone help me point out
the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7293c299-16ad-420f-bcad-2e0d80681b17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

any thoughts anyone ?

On Wednesday, June 11, 2014 11:15:18 PM UTC-7, Srinivasan Ramaswamy wrote:

I would like to exclude some documents belonging to certain category from
the results only for certain search queries. I have a ES client layer where
i am thinking of implementing this logic as a "not" filter depending on the
search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for other
keywords) but filter out few categories from the result. As ES returns the
whole nested document even if only one nested document matches the query,
my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]
Here is the query i tried but it doesn't work. Can anyone help me point
out the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/45fbf85d-4d29-4222-a72a-bf0a04d9a26d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Currently not possible. Elasticsearch will return all the nested documents
as long as one of the nested documents satisfies the query.

The issue is my personal #1 feature requested. Frustrating considering
there has been a working implementation since version 0.90.5. 1.0, 1.1, 1.2
and still nothing.

--
Ivan

On Thu, Jun 12, 2014 at 2:17 PM, Srinivasan Ramaswamy ursvasan@gmail.com
wrote:

any thoughts anyone ?

On Wednesday, June 11, 2014 11:15:18 PM UTC-7, Srinivasan Ramaswamy wrote:

I would like to exclude some documents belonging to certain category from
the results only for certain search queries. I have a ES client layer where
i am thinking of implementing this logic as a "not" filter depending on the
search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for
other keywords) but filter out few categories from the result. As ES
returns the whole nested document even if only one nested document matches
the query, my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]
Here is the query i tried but it doesn't work. Can anyone help me point
out the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45fbf85d-4d29-4222-a72a-bf0a04d9a26d%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi Ivan

Thanks for your reply. Yeah, I do understand that currently elasticsearch
returns the whole nested doc.
Can you help me how can i get the negative query with multiple categories
working ?

Thanks
Srini

On Fri, Jun 13, 2014 at 10:58 AM, Ivan Brusic ivan@brusic.com wrote:

Currently not possible. Elasticsearch will return all the nested documents
as long as one of the nested documents satisfies the query.

https://github.com/elasticsearch/elasticsearch/issues/3022

The issue is my personal #1 feature requested. Frustrating considering
there has been a working implementation since version 0.90.5. 1.0, 1.1, 1.2
and still nothing.

--
Ivan

On Thu, Jun 12, 2014 at 2:17 PM, Srinivasan Ramaswamy ursvasan@gmail.com
wrote:

any thoughts anyone ?

On Wednesday, June 11, 2014 11:15:18 PM UTC-7, Srinivasan Ramaswamy wrote:

I would like to exclude some documents belonging to certain category
from the results only for certain search queries. I have a ES client layer
where i am thinking of implementing this logic as a "not" filter depending
on the search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for
other keywords) but filter out few categories from the result. As ES
returns the whole nested document even if only one nested document matches
the query, my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]
Here is the query i tried but it doesn't work. Can anyone help me
point out the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45fbf85d-4d29-4222-a72a-bf0a04d9a26d%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Fqt70gBtypQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL1MvVyOvd_YDi92z_fH9-OE6VJTcOP-Q4E-BvKj94FSkvEJOw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

I jumped the gun when I thought I realized the issue.

You listed your expected result, but not your actual result. Are you
actually using nested documents? If so, you would need to use nested
queries/filters:

Are you looking for a workaround for the issue referenced? You would either
need to do the extra filtering on the client side or push all the nested
values into the parent and query on that field.

--
Ivan

On Fri, Jun 13, 2014 at 11:52 AM, Srinivasan Ramaswamy ursvasan@gmail.com
wrote:

Hi Ivan

Thanks for your reply. Yeah, I do understand that currently elasticsearch
returns the whole nested doc.
Can you help me how can i get the negative query with multiple categories
working ?

Thanks
Srini

On Fri, Jun 13, 2014 at 10:58 AM, Ivan Brusic ivan@brusic.com wrote:

Currently not possible. Elasticsearch will return all the nested
documents as long as one of the nested documents satisfies the query.

https://github.com/elasticsearch/elasticsearch/issues/3022

The issue is my personal #1 feature requested. Frustrating considering
there has been a working implementation since version 0.90.5. 1.0, 1.1, 1.2
and still nothing.

--
Ivan

On Thu, Jun 12, 2014 at 2:17 PM, Srinivasan Ramaswamy <ursvasan@gmail.com

wrote:

any thoughts anyone ?

On Wednesday, June 11, 2014 11:15:18 PM UTC-7, Srinivasan Ramaswamy
wrote:

I would like to exclude some documents belonging to certain category
from the results only for certain search queries. I have a ES client layer
where i am thinking of implementing this logic as a "not" filter depending
on the search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for
other keywords) but filter out few categories from the result. As ES
returns the whole nested document even if only one nested document matches
the query, my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]
Here is the query i tried but it doesn't work. Can anyone help me
point out the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45fbf85d-4d29-4222-a72a-bf0a04d9a26d%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Fqt70gBtypQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL1MvVyOvd_YDi92z_fH9-OE6VJTcOP-Q4E-BvKj94FSkvEJOw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL1MvVyOvd_YDi92z_fH9-OE6VJTcOP-Q4E-BvKj94FSkvEJOw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA6t7Admm4foKB8C6f8rFjt-4wrb0TwTJ-ft3xbvZuZNg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Yeah, I forgot to include my actual result. My "not" filter was not working
at all. I got all the 3 designs back: 100, 101 and 102.

I followed the syntax in the link you sent and it worked :slight_smile: I tried similar
syntax a few times before i posted the question, but i didn't have a
"filter" clause inside the "nested" clause (basically two "filter"
clauses). That made the trick !

Thanks a lot !
Srini

On Tue, Jun 17, 2014 at 4:46 PM, Ivan Brusic ivan@brusic.com wrote:

I jumped the gun when I thought I realized the issue.

You listed your expected result, but not your actual result. Are you
actually using nested documents? If so, you would need to use nested
queries/filters:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Are you looking for a workaround for the issue referenced? You would
either need to do the extra filtering on the client side or push all the
nested values into the parent and query on that field.

--
Ivan

On Fri, Jun 13, 2014 at 11:52 AM, Srinivasan Ramaswamy <ursvasan@gmail.com

wrote:

Hi Ivan

Thanks for your reply. Yeah, I do understand that currently elasticsearch
returns the whole nested doc.
Can you help me how can i get the negative query with multiple categories
working ?

Thanks
Srini

On Fri, Jun 13, 2014 at 10:58 AM, Ivan Brusic ivan@brusic.com wrote:

Currently not possible. Elasticsearch will return all the nested
documents as long as one of the nested documents satisfies the query.

https://github.com/elasticsearch/elasticsearch/issues/3022

The issue is my personal #1 feature requested. Frustrating considering
there has been a working implementation since version 0.90.5. 1.0, 1.1, 1.2
and still nothing.

--
Ivan

On Thu, Jun 12, 2014 at 2:17 PM, Srinivasan Ramaswamy <
ursvasan@gmail.com> wrote:

any thoughts anyone ?

On Wednesday, June 11, 2014 11:15:18 PM UTC-7, Srinivasan Ramaswamy
wrote:

I would like to exclude some documents belonging to certain category
from the results only for certain search queries. I have a ES client layer
where i am thinking of implementing this logic as a "not" filter depending
on the search query. Let me give an example.

sample index

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 101
tags: ["brown", "dog"]
caption : little brown dog
products : [ {productId: "202", category: 3} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]

products is a nested field inside each design.

I would like to write a query to get all matches for "dog", (not for
other keywords) but filter out few categories from the result. As ES
returns the whole nested document even if only one nested document matches
the query, my expected result is

designId: 100
tags: ["dog", "cute"]
caption : cute dog in the garden
products : [ { productId: "200", category: 1}, {productId: "201",
category: 2} ]

designId: 102
tags: ["black", "dog"]
caption : little black dog
products : [ { productId: "202", category: 4}, {productId: "203",
category: 5} ]
Here is the query i tried but it doesn't work. Can anyone help me
point out the mistake ?

GET /_search/
{
"query": {
"filtered": {
"filter": {
"and": [
{
"not": {
"term": {
"category": 1
}
}
},
{
"not": {
"term": {
"category": 3
}
}
}
]

     },
     "query": {
        "multi_match": {
           "query": "dog",
           "fields": [
              "tags",
              "caption"
           ],
           "minimum_should_match": "50%"
        }
     }
  }

}
}

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/45fbf85d-4d29-4222-a72a-bf0a04d9a26d%40googlegroups.com
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Fqt70gBtypQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQAfwARsZ7uGKkBf%2BH10jhrdw4dr5nxvHEK_FDUwQv%2BpQw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL1MvVyOvd_YDi92z_fH9-OE6VJTcOP-Q4E-BvKj94FSkvEJOw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL1MvVyOvd_YDi92z_fH9-OE6VJTcOP-Q4E-BvKj94FSkvEJOw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Fqt70gBtypQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA6t7Admm4foKB8C6f8rFjt-4wrb0TwTJ-ft3xbvZuZNg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA6t7Admm4foKB8C6f8rFjt-4wrb0TwTJ-ft3xbvZuZNg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL1MvVxLCuiYz1UNtxdEGUZGN5hnLNS%2BM8LpGoXpc7YMt-TqYg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.