Neither query nor filter

Elastic Stack Elasticsearch
1

Hi list,

Let's put I have a simple 'index' with a simple 'type' mapping animals like
this:

{
"name": "dog",
"type": "mammal"
}

{
"name": "dolphin",
"type": "mammal"
}

{
"name": "tiger",
"type": "mammal"
}

{
"name": "crocodile",
"type": "reptile"
}

{
"name": "lizard",
"type": "reptile"
}

Now I want to query something like (may make little sense, but it's the
concept what I'm trying to show here):

"Find all animals which are either mamals and its name starts with 'do', or
reptiles and its name ends with 'le'"

So analyzing this query, I need an OR with two conditions, each one also
with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD I
just cannot use an AND to force several criteria at a time.

On the other hand, if I use a filtered query, and use the AND/OR mechanism
from the filters, then I cannot use the 'field' query (or anything which
requires full-text search capabilities), as the filters do not provide such
functionality.

So what should I do? I'm afraid this is a very simple use case, but I'm
struggling to figure out the best way to handle it.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi Enrique

So analyzing this query, I need an OR with two conditions, each one
also with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD
I just cannot use an AND to force several criteria at a time.

This'll work:

{
"query" : {
"bool" : {
"should" : [
{
"bool" : {
"must" : [
{ "field" : { "name" : "do*" }},
{ "match" : { "type" : "mammal"}}
]
}
},
{
"bool" : {
"must" : [
{ "field" : { "name" : "le*" }},
{ "match" : { "type" : "reptile"}}
]
}
}
]
}
}
}

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Another easier way, which I am using in my project is use query_string
format

{
"query" : {
"query_string" : {
"query" :
"(((type:mammal)AND(name:do\*))OR((type:reptile)AND(name:\*le)))"
}
}
}

-Maaz

On Thursday, March 28, 2013 5:19:52 PM UTC+5, Enrique Medina Montenegro
wrote:

Hi list,

Let's put I have a simple 'index' with a simple 'type' mapping animals
like this:

{
"name": "dog",
"type": "mammal"
}

{
"name": "dolphin",
"type": "mammal"
}

{
"name": "tiger",
"type": "mammal"
}

{
"name": "crocodile",
"type": "reptile"
}

{
"name": "lizard",
"type": "reptile"
}

Now I want to query something like (may make little sense, but it's the
concept what I'm trying to show here):

"Find all animals which are either mamals and its name starts with 'do',
or reptiles and its name ends with 'le'"

So analyzing this query, I need an OR with two conditions, each one also
with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD I
just cannot use an AND to force several criteria at a time.

On the other hand, if I use a filtered query, and use the AND/OR mechanism
from the filters, then I cannot use the 'field' query (or anything which
requires full-text search capabilities), as the filters do not provide such
functionality.

So what should I do? I'm afraid this is a very simple use case, but I'm
struggling to figure out the best way to handle it.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Thanks Clinton.

That nested bool query stuff is missing in the docs...

El jueves, 28 de marzo de 2013 13:47:52 UTC+1, Clinton Gormley escribió:

Hi Enrique

So analyzing this query, I need an OR with two conditions, each one
also with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD
I just cannot use an AND to force several criteria at a time.

This'll work:

{
"query" : {
"bool" : {
"should" : [
{
"bool" : {
"must" : [
{ "field" : { "name" : "do*" }},
{ "match" : { "type" : "mammal"}}
]
}
},
{
"bool" : {
"must" : [
{ "field" : { "name" : "le*" }},
{ "match" : { "type" : "reptile"}}
]
}
}
]
}
}
}

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Maaz,

I was not choosing your approach because as far as I understand, it would
analyze and tokenize each field at query time, therefore potentially a
bottleneck if too much data...

Actually I'm using edge-ngrams to analyze it at index time, but your
feedback is very interesting as well. Are you facing performance issues
with your approach and huge amounts of data?

Regards.

El jueves, 28 de marzo de 2013 14:04:34 UTC+1, Maaz Bin Tariq escribió:

Another easier way, which I am using in my project is use query_string
format

{
"query" : {
"query_string" : {
"query" :
"(((type:mammal)AND(name:do\*))OR((type:reptile)AND(name:\*le)))"
}
}
}

-Maaz

On Thursday, March 28, 2013 5:19:52 PM UTC+5, Enrique Medina Montenegro
wrote:

Hi list,

Let's put I have a simple 'index' with a simple 'type' mapping animals
like this:

{
"name": "dog",
"type": "mammal"
}

{
"name": "dolphin",
"type": "mammal"
}

{
"name": "tiger",
"type": "mammal"
}

{
"name": "crocodile",
"type": "reptile"
}

{
"name": "lizard",
"type": "reptile"
}

Now I want to query something like (may make little sense, but it's the
concept what I'm trying to show here):

"Find all animals which are either mamals and its name starts with 'do',
or reptiles and its name ends with 'le'"

So analyzing this query, I need an OR with two conditions, each one also
with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD I
just cannot use an AND to force several criteria at a time.

On the other hand, if I use a filtered query, and use the AND/OR
mechanism from the filters, then I cannot use the 'field' query (or
anything which requires full-text search capabilities), as the filters do
not provide such functionality.

So what should I do? I'm afraid this is a very simple use case, but I'm
struggling to figure out the best way to handle it.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Clinton,

BTW, is this the only approach, or could I use filters and then filter
queries for "full-text" requirements?

Thanks.

El jueves, 28 de marzo de 2013 13:47:52 UTC+1, Clinton Gormley escribió:

Hi Enrique

So analyzing this query, I need an OR with two conditions, each one
also with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD
I just cannot use an AND to force several criteria at a time.

This'll work:

{
"query" : {
"bool" : {
"should" : [
{
"bool" : {
"must" : [
{ "field" : { "name" : "do*" }},
{ "match" : { "type" : "mammal"}}
]
}
},
{
"bool" : {
"must" : [
{ "field" : { "name" : "le*" }},
{ "match" : { "type" : "reptile"}}
]
}
}
]
}
}
}

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

On Thu, 2013-03-28 at 07:07 -0700, Enrique Medina Montenegro wrote:

Clinton,

BTW, is this the only approach, or could I use filters and then filter
queries for "full-text" requirements?

Yes you can combine them, eg:

{
"query" : {
"bool" : {
"should" : [
{
"filtered" : {
"filter" : {
"term" : {
"type" : "mammal"
}
},
"query" : {
"field" : {
"name" : "do*"
}
}
}
},
{
"filtered" : {
"filter" : {
"term" : {
"type" : "reptile"
}
},
"query" : {
"field" : {
"name" : "le*"
}
}
}
}
]
}
}
}

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

As far as my understanding it would not analyze all the fields, yes
tokenize (grouping) it will definitely do using lucene grouping and query
format support which I guess is the elasticsearch base.

I am not facing any performance issues yet on few million records but not
sure for very large data set. May be an expert can help

@Clinton what are the possible issues (performance or others) in using my
approach. As it will also help me alot

Thanks
-Maaz

On Thursday, March 28, 2013 6:54:57 PM UTC+5, Enrique Medina Montenegro
wrote:

Maaz,

I was not choosing your approach because as far as I understand, it would
analyze and tokenize each field at query time, therefore potentially a
bottleneck if too much data...

Actually I'm using edge-ngrams to analyze it at index time, but your
feedback is very interesting as well. Are you facing performance issues
with your approach and huge amounts of data?

Regards.

El jueves, 28 de marzo de 2013 14:04:34 UTC+1, Maaz Bin Tariq escribió:

Another easier way, which I am using in my project is use query_string
format

{
"query" : {
"query_string" : {
"query" :
"(((type:mammal)AND(name:do\*))OR((type:reptile)AND(name:\*le)))"
}
}
}

-Maaz

On Thursday, March 28, 2013 5:19:52 PM UTC+5, Enrique Medina Montenegro
wrote:

Hi list,

Let's put I have a simple 'index' with a simple 'type' mapping animals
like this:

{
"name": "dog",
"type": "mammal"
}

{
"name": "dolphin",
"type": "mammal"
}

{
"name": "tiger",
"type": "mammal"
}

{
"name": "crocodile",
"type": "reptile"
}

{
"name": "lizard",
"type": "reptile"
}

Now I want to query something like (may make little sense, but it's the
concept what I'm trying to show here):

"Find all animals which are either mamals and its name starts with 'do',
or reptiles and its name ends with 'le'"

So analyzing this query, I need an OR with two conditions, each one also
with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD I
just cannot use an AND to force several criteria at a time.

On the other hand, if I use a filtered query, and use the AND/OR
mechanism from the filters, then I cannot use the 'field' query (or
anything which requires full-text search capabilities), as the filters do
not provide such functionality.

So what should I do? I'm afraid this is a very simple use case, but I'm
struggling to figure out the best way to handle it.

Thanks!

On Thursday, March 28, 2013 6:54:57 PM UTC+5, Enrique Medina Montenegro
wrote:

Maaz,

I was not choosing your approach because as far as I understand, it would
analyze and tokenize each field at query time, therefore potentially a
bottleneck if too much data...

Actually I'm using edge-ngrams to analyze it at index time, but your
feedback is very interesting as well. Are you facing performance issues
with your approach and huge amounts of data?

Regards.

El jueves, 28 de marzo de 2013 14:04:34 UTC+1, Maaz Bin Tariq escribió:

Another easier way, which I am using in my project is use query_string
format

{
"query" : {
"query_string" : {
"query" :
"(((type:mammal)AND(name:do\*))OR((type:reptile)AND(name:\*le)))"
}
}
}

-Maaz

On Thursday, March 28, 2013 5:19:52 PM UTC+5, Enrique Medina Montenegro
wrote:

Hi list,

Let's put I have a simple 'index' with a simple 'type' mapping animals
like this:

{
"name": "dog",
"type": "mammal"
}

{
"name": "dolphin",
"type": "mammal"
}

{
"name": "tiger",
"type": "mammal"
}

{
"name": "crocodile",
"type": "reptile"
}

{
"name": "lizard",
"type": "reptile"
}

Now I want to query something like (may make little sense, but it's the
concept what I'm trying to show here):

"Find all animals which are either mamals and its name starts with 'do',
or reptiles and its name ends with 'le'"

So analyzing this query, I need an OR with two conditions, each one also
with an AND inside:

OR {
{
AND {
"field": { "name": "do*" }
},
{
"term": { "type": "mammal" }
}
}
},
{
AND {
"field": { "name": "*le" }
},
{
"term": { "type": "reptile" }
}
}
}
}

Using a query I don't have AND/OR, but SHOULD; however, within SHOULD I
just cannot use an AND to force several criteria at a time.

On the other hand, if I use a filtered query, and use the AND/OR
mechanism from the filters, then I cannot use the 'field' query (or
anything which requires full-text search capabilities), as the filters do
not provide such functionality.

So what should I do? I'm afraid this is a very simple use case, but I'm
struggling to figure out the best way to handle it.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Hi Maaz

On Thu, 2013-03-28 at 21:42 -0700, Maaz Bin Tariq wrote:

As far as my understanding it would not analyze all the fields, yes
tokenize (grouping) it will definitely do using lucene grouping and
query format support which I guess is the elasticsearch base.

The query_string format is prone to errors. You have to be very careful
how you format the query - if you nest things incorrectly you can end up
with the wrong results, and it can be tricky to see WHY they are wrong.
Any syntax error will result in no query being run at all, eg:

"foo - bar" or "http:"

Also, the query string query uses only queries, which aren't cached, and
can affect the scoring even when you don't want it to.

The query DSL gives you much more control over your searches. You can
use filters where appropriate to benefit from caching and to ensure that
they don't interfere with scoring. The fact that they don't NEED to
score also improves performance.

There is more of a learning curve to getting started with the query DSL,
but it is worth the effort.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

hi Clint,

The query string is easily readable when using brackets and especial
characters (+,-,{ etc) need to be escaped (link1 below) to get the required
results and that's how lucene and Solr works.

The major disadvantage of it is results are scored on each search, so it
will not be cached and if we apply filters it would not get the cache
benefit, right?
and as per @Enrique would it analyze and tokenize each field at query time?
will that a performance hit?

I will also try more in in learning Query DSL as it very broad :slight_smile:

link1:
org.apache.lucene.queryparser.classic (Lucene 4.2.0 API)

Thanks
-Maaz

On Friday, March 29, 2013 3:35:53 PM UTC+5, Clinton Gormley wrote:

Hi Maaz

On Thu, 2013-03-28 at 21:42 -0700, Maaz Bin Tariq wrote:

As far as my understanding it would not analyze all the fields, yes
tokenize (grouping) it will definitely do using lucene grouping and
query format support which I guess is the elasticsearch base.

The query_string format is prone to errors. You have to be very careful
how you format the query - if you nest things incorrectly you can end up
with the wrong results, and it can be tricky to see WHY they are wrong.
Any syntax error will result in no query being run at all, eg:

"foo - bar" or "http:"

Also, the query string query uses only queries, which aren't cached, and
can affect the scoring even when you don't want it to.

The query DSL gives you much more control over your searches. You can
use filters where appropriate to benefit from caching and to ensure that
they don't interfere with scoring. The fact that they don't NEED to
score also improves performance.

There is more of a learning curve to getting started with the query DSL,
but it is worth the effort.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

11

Hi Maaz

The query string is easily readable when using brackets and especial
characters (+,-,{ etc) need to be escaped

don't forget spaces, ~ : | etc etc

(link1 below) to get the required results and that's how lucene and
Solr works.

Readable is in the eye of the beholder :slight_smile:

The major disadvantage of it is results are scored on each search, so
it will not be cached and if we apply filters it would not get the
cache benefit, right?

Yes

and as per @Enrique would it analyze and tokenize each field at query
time? will that a performance hit?

Not really. The query string query has its own parser, so does a tiny
bit more work, but not so you'd notice.

I will also try more in in learning Query DSL as it very broad :slight_smile:

It's quite old now, but is still useful to get started:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

12

Great help, Thanks

On Fri, Mar 29, 2013 at 4:52 PM, Clinton Gormley clint@traveljury.comwrote:

Hi Maaz

The query string is easily readable when using brackets and especial
characters (+,-,{ etc) need to be escaped

don't forget spaces, ~ : | etc etc

(link1 below) to get the required results and that's how lucene and
Solr works.

Readable is in the eye of the beholder :slight_smile:

The major disadvantage of it is results are scored on each search, so
it will not be cached and if we apply filters it would not get the
cache benefit, right?

Yes

and as per @Enrique would it analyze and tokenize each field at query
time? will that a performance hit?

Not really. The query string query has its own parser, so does a tiny
bit more work, but not so you'd notice.

I will also try more in in learning Query DSL as it very broad :slight_smile:

It's quite old now, but is still useful to get started:

https://speakerdeck.com/clintongormley/terms-of-endearment-the-elasticsearch-query-dsl-explained

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/p1CTUMVBPVw/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.