Problem in filtering data based on individual fields


(Srinath C) #1

Hi,
I'm a newbie to elasticsearch and I have been trying to understand the
Query DSL and am having some difficulty in querying data using filters.

Records are being added to an index with a structure as below:
{ "applicationId": "TestApp", "timestamp": 1393649305868, "value": 20 }
{ "applicationId": "App", "timestamp": 1393649305868, "value": 20 }

After adding sufficient number of records, am trying to query back the
data using the query (Path /{index}/{type}/_search ):

  • { "query": { "match_all": { } }, "filter": { "bool": { "must": [ {
    "term" : { "applicationId" : "TestApp" } } ] } }, "size": 10 }*

    But no records match the query. I looked at the way kibana is creating
    queries for the same and it looks pretty complex (attached).

{"took":6,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Can anyone reason out what is wrong with the above filter based query
and can there be a simpler query than what kibana uses(attached)?

Thanks,
Srinath.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8dd18c69-1b5d-406a-845b-d3a6858b0293%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #2

Try testapp lowercase.
Term query is not analyzed.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 mars 2014 à 04:00, Srinath C srinath.c@gmail.com a écrit :

Hi,
I'm a newbie to elasticsearch and I have been trying to understand the Query DSL and am having some difficulty in querying data using filters.

Records are being added to an index with a structure as below:
{ "applicationId": "TestApp", "timestamp": 1393649305868, "value": 20 }
{ "applicationId": "App", "timestamp": 1393649305868, "value": 20 }

After adding sufficient number of records, am trying to query back the data using the query (Path /{index}/{type}/_search ):
{ "query": { "match_all": { } }, "filter": { "bool": { "must": [ { "term" : { "applicationId" : "TestApp" } } ] } }, "size": 10 }

But no records match the query. I looked at the way kibana is creating queries for the same and it looks pretty complex (attached).
{"took":6,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Can anyone reason out what is wrong with the above filter based query and can there be a simpler query than what kibana uses(attached)?

Thanks,
Srinath.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8dd18c69-1b5d-406a-845b-d3a6858b0293%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<kibana.json>

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E87CFBD6-7509-4E3C-868A-76D171CE83A6%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.


(Srinath C) #3

Thanks David, it indeed works if I use "testapp" instead of "TestApp"!!
Can you point me to some links to understand this better? What does it mean
by saying "Term query is not analyzed"?
If my values are case-sensitive which types of filter are suitable?

On Mon, Mar 3, 2014 at 12:57 PM, David Pilato david@pilato.fr wrote:

Try testapp lowercase.
Term query is not analyzed.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 mars 2014 à 04:00, Srinath C srinath.c@gmail.com a écrit :

Hi,
I'm a newbie to elasticsearch and I have been trying to understand the
Query DSL and am having some difficulty in querying data using filters.

Records are being added to an index with a structure as below:
{ "applicationId": "TestApp", "timestamp": 1393649305868, "value": 20
}

{ "applicationId": "App", "timestamp": 1393649305868, "value": 20 }

After adding sufficient number of records, am trying to query back the
data using the query (Path /{index}/{type}/_search ):

  • { "query": { "match_all": { } }, "filter": { "bool": { "must": [ {
    "term" : { "applicationId" : "TestApp" } } ] } }, "size": 10 }*

    But no records match the query. I looked at the way kibana is creating
    queries for the same and it looks pretty complex (attached).

{"took":6,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Can anyone reason out what is wrong with the above filter based query
and can there be a simpler query than what kibana uses(attached)?

Thanks,
Srinath.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8dd18c69-1b5d-406a-845b-d3a6858b0293%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
<kibana.json>

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_W3HSc-2Lds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E87CFBD6-7509-4E3C-868A-76D171CE83A6%40pilato.fr
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHhx-GK5jWrWEauW4ATSUzhKKocwr_Vhk8MEy%2B6A5BNxTjLjnw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #4

By default, your content is processed by analyzers: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html
The default one will analyze "TestApp" to [ "testapp" ].

That's what is going to be indexed in the inverted index.

When searching, you search in the index not in source docs. Searching for "TestApp" won't match "testapp".
TermFilter is not analyzed. That means your query won't be analyzed by the same analyzer used at index time.

MatchQuery is for example analyzed. So searching for "TESTApp" will work...

Hope this helps

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 3 mars 2014 à 10:20:04, Srinath C (srinath.c@gmail.com) a écrit:

Thanks David, it indeed works if I use "testapp" instead of "TestApp"!!
Can you point me to some links to understand this better? What does it mean by saying "Term query is not analyzed"?
If my values are case-sensitive which types of filter are suitable?

On Mon, Mar 3, 2014 at 12:57 PM, David Pilato david@pilato.fr wrote:
Try testapp lowercase.
Term query is not analyzed.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 mars 2014 à 04:00, Srinath C srinath.c@gmail.com a écrit :

Hi,
I'm a newbie to elasticsearch and I have been trying to understand the Query DSL and am having some difficulty in querying data using filters.

Records are being added to an index with a structure as below:
{ "applicationId": "TestApp", "timestamp": 1393649305868, "value": 20 }
{ "applicationId": "App", "timestamp": 1393649305868, "value": 20 }

After adding sufficient number of records, am trying to query back the data using the query (Path /{index}/{type}/_search ):
{ "query": { "match_all": { } }, "filter": { "bool": { "must": [ { "term" : { "applicationId" : "TestApp" } } ] } }, "size": 10 }

But no records match the query. I looked at the way kibana is creating queries for the same and it looks pretty complex (attached).
{"took":6,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Can anyone reason out what is wrong with the above filter based query and can there be a simpler query than what kibana uses(attached)?

Thanks,
Srinath.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8dd18c69-1b5d-406a-845b-d3a6858b0293%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
<kibana.json>

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/_W3HSc-2Lds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/E87CFBD6-7509-4E3C-868A-76D171CE83A6%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHhx-GK5jWrWEauW4ATSUzhKKocwr_Vhk8MEy%2B6A5BNxTjLjnw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.5314555a.6b8b4567.50c7%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/groups/opt_out.


(Srinath C) #5

Thanks David, it explains a lot. Will dig deeper using these links.

On Mon, Mar 3, 2014 at 3:41 PM, David Pilato david@pilato.fr wrote:

By default, your content is processed by analyzers:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/analysis-standard-analyzer.html
The default one will analyze "TestApp" to [ "testapp" ].

That's what is going to be indexed in the inverted index.

When searching, you search in the index not in source docs. Searching for
"TestApp" won't match "testapp".
TermFilter is not analyzed. That means your query won't be analyzed by the
same analyzer used at index time.

MatchQuery is for example analyzed. So searching for "TESTApp" will work...

Hope this helps

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfrhttps://twitter.com/elasticsearchfr

Le 3 mars 2014 à 10:20:04, Srinath C (srinath.c@gmail.com) a écrit:

Thanks David, it indeed works if I use "testapp" instead of "TestApp"!!
Can you point me to some links to understand this better? What does it
mean by saying "Term query is not analyzed"?
If my values are case-sensitive which types of filter are suitable?

On Mon, Mar 3, 2014 at 12:57 PM, David Pilato david@pilato.fr wrote:

Try testapp lowercase.
Term query is not analyzed.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 3 mars 2014 à 04:00, Srinath C srinath.c@gmail.com a écrit :

Hi,
I'm a newbie to elasticsearch and I have been trying to understand the
Query DSL and am having some difficulty in querying data using filters.

Records are being added to an index with a structure as below:
{ "applicationId": "TestApp", "timestamp": 1393649305868, "value": 20
}

{ "applicationId": "App", "timestamp": 1393649305868, "value": 20 }

After adding sufficient number of records, am trying to query back

the data using the query (Path /{index}/{type}/_search ):

  • { "query": { "match_all": { } }, "filter": { "bool": { "must": [ {
    "term" : { "applicationId" : "TestApp" } } ] } }, "size": 10 }*

    But no records match the query. I looked at the way kibana is
    creating queries for the same and it looks pretty complex (attached).

{"took":6,"timed_out":false,"_shards":{"total":5,"successful":5,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

Can anyone reason out what is wrong with the above filter based query
and can there be a simpler query than what kibana uses(attached)?

Thanks,
Srinath.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8dd18c69-1b5d-406a-845b-d3a6858b0293%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
<kibana.json>

You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_W3HSc-2Lds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/E87CFBD6-7509-4E3C-868A-76D171CE83A6%40pilato.fr.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHhx-GK5jWrWEauW4ATSUzhKKocwr_Vhk8MEy%2B6A5BNxTjLjnw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/_W3HSc-2Lds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/etPan.5314555a.6b8b4567.50c7%40MacBook-Air-de-David.local
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHhx-GKeF6hckYkJutUqZtpV%2BxS1ry_n%3DoWSxfbqsuDrykZr4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6