Time range search fails


(subin) #1

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format

  • Oct 3 22:30:55. I have this mappinghttps://gist.github.com/osssubb/6774727 as
    a template in ES. You may see that @fields.date mapping is set as - "date":
    { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index":
    "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries.
Below is the query I execute, on Kibana 0.2.0, to achieve range-searches,
but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3
21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #2

Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format - Oct 3 22:30:55. I have this mapping as a template in ES. You may see that @fields.date mapping is set as - "date": { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index": "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries. Below is the query I execute, on Kibana 0.2.0, to achieve range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #3

Hi David,

It doesn't return any result at all, not even the logs which have those
dates as time-stamps.

Re,


Subin
*
*

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:

Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format

  • Oct 3 22:30:55. I have this mappinghttps://gist.github.com/osssubb/6774727 as
    a template in ES. You may see that @fields.date mapping is set as - "date":
    { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index":
    "analyzed" },

@fields.date is also the field I'll need to execute the date-range
queries. Below is the query I execute, on Kibana 0.2.0, to achieve
range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3
21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #4

Try

@fields.date:["2013-10-02" TO "2013-10-03"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 06:28, subin ks ksubins321@gmail.com a écrit :

Hi David,

It doesn't return any result at all, not even the logs which have those dates as time-stamps.

Re,


Subin

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:

Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format - Oct 3 22:30:55. I have this mapping as a template in ES. You may see that @fields.date mapping is set as - "date": { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index": "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries. Below is the query I execute, on Kibana 0.2.0, to achieve range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(subin) #5

ES errors out with this format. An excerpt from the error stack:

at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.ElasticSearchParseException: failed to parse
date field [2013-10-02], tried both date format [MMM dd HH:mm:ss||MMM d
HH:mm:ss], and timestamp number
at
org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:194)
at
org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:49)
at
org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:23)
at
org.elasticsearch.index.mapper.core.DateFieldMapper.rangeQuery(DateFieldMapper.java:310)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuerySingle(MapperQueryParser.java:377)
at
org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuery(MapperQueryParser.java:331)
at
org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:441)
at
org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:257)
at
org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:181)
at
org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:170)
at
org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:121)
at
org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:214)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at
org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at
org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:255)
at
org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:549)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "2013-10-02"
at
org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at
org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:188)
... 30 more


Subin
*
*

On Fri, Oct 4, 2013 at 10:42 AM, David Pilato david@pilato.fr wrote:

Try

@fields.date:["2013-10-02" TO "2013-10-03"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 06:28, subin ks ksubins321@gmail.com a écrit :

Hi David,

It doesn't return any result at all, not even the logs which have those
dates as time-stamps.

Re,


Subin
*
*

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:

Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format

  • Oct 3 22:30:55. I have this mappinghttps://gist.github.com/osssubb/6774727 as
    a template in ES. You may see that @fields.date mapping is set as - "date":
    { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index":
    "analyzed" },

@fields.date is also the field I'll need to execute the date-range
queries. Below is the query I execute, on Kibana 0.2.0, to achieve
range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct
3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #6

I was expecting such a trace.
So that's strange the previous form did not work.

I have no more idea yet.

Could you reproduce it in a curl recreation (delete index, create mapping, put doc in between thoses dates, refresh, search with query string)? And Gist it.

It would help to understand. I'm probably missing something.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 07:21, subin ks ksubins321@gmail.com a écrit :

ES errors out with this format. An excerpt from the error stack:

at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.ElasticSearchParseException: failed to parse date field [2013-10-02], tried both date format [MMM dd HH:mm:ss||MMM d HH:mm:ss], and timestamp number
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:194)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:49)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:23)
at org.elasticsearch.index.mapper.core.DateFieldMapper.rangeQuery(DateFieldMapper.java:310)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuerySingle(MapperQueryParser.java:377)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuery(MapperQueryParser.java:331)
at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:441)
at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:257)
at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:181)
at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:170)
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:121)
at org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:214)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:255)
at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:549)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "2013-10-02"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:188)
... 30 more


Subin

On Fri, Oct 4, 2013 at 10:42 AM, David Pilato david@pilato.fr wrote:

Try

@fields.date:["2013-10-02" TO "2013-10-03"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 06:28, subin ks ksubins321@gmail.com a écrit :

Hi David,

It doesn't return any result at all, not even the logs which have those dates as time-stamps.

Re,


Subin

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:

Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format - Oct 3 22:30:55. I have this mapping as a template in ES. You may see that @fields.date mapping is set as - "date": { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index": "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries. Below is the query I execute, on Kibana 0.2.0, to achieve range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jun Ohtani) #7

Hi,

I am interested this case.
And I try to send range query to _validate api.

curl -XPOST 'localhost:9200/test/logstash/_validate/query?pretty&explain=true' -d '
{
"query_string": {
"query": "@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]"
}
}'

and I recieve strange response.

{
"valid" : true,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"explanations" : [ {
"index" : "test",
"valid" : true,
"explanation" : "@fields.date:[970524000000 TO 23835600999]"
} ]

I use ES 0.90.5.

Does this information helpful?


Jun Ohtani
johtani@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

On 2013/10/04, at 14:46, David Pilato david@pilato.fr wrote:

I was expecting such a trace.
So that's strange the previous form did not work.

I have no more idea yet.

Could you reproduce it in a curl recreation (delete index, create mapping, put doc in between thoses dates, refresh, search with query string)? And Gist it.

It would help to understand. I'm probably missing something.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 07:21, subin ks ksubins321@gmail.com a écrit :

ES errors out with this format. An excerpt from the error stack:

at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.ElasticSearchParseException: failed to parse date field [2013-10-02], tried both date format [MMM dd HH:mm:ss||MMM d HH:mm:ss], and timestamp number
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:194)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:49)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:23)
at org.elasticsearch.index.mapper.core.DateFieldMapper.rangeQuery(DateFieldMapper.java:310)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuerySingle(MapperQueryParser.java:377)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuery(MapperQueryParser.java:331)
at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:441)
at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:257)
at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:181)
at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:170)
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:121)
at org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:214)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:255)
at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:549)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "2013-10-02"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:188)
... 30 more


Subin

On Fri, Oct 4, 2013 at 10:42 AM, David Pilato david@pilato.fr wrote:
Try

@fields.date:["2013-10-02" TO "2013-10-03"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 06:28, subin ks ksubins321@gmail.com a écrit :

Hi David,

It doesn't return any result at all, not even the logs which have those dates as time-stamps.

Re,


Subin

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:
Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format - Oct 3 22:30:55. I have this mapping as a template in ES. You may see that @fields.date mapping is set as - "date": { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index": "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries. Below is the query I execute, on Kibana 0.2.0, to achieve range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jun Ohtani) #8

Hi,

I'm debuging this case.

Maybe, this problem cause that date string is not include year part.
QueryStringParser parse query, then to create range query process following code.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/index/mapper/core/DateFieldMapper.java#L321

Range query from part( this case "Oct 2 22:00:00") is parsed dateMathParser.parse() method.
Ant Range query to part( this case "Oct 3 21:00:00") is parsed dateMathParser.parseUpperInclusive() method.(following link)

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/joda/DateMathParser.java#L199

This method convert "Oct 3 21:00:00" to "1970-10-03T21:00:00.999Z".

I have 2 ideas to avoid this problem .

  1. You should register document to include year.
    But this idea require re-index all data.

  2. Set "true" to "index.mapping.date.parse_upper_inclusive" property.
    To set this property execute Following command.
    a. close index.
    curl -XPOST 'localhost:9200/test/_close'
    b. set property
    curl -XPUT 'localhost:9200/test/_settings' -d '
    { "index":{ "mapping":{ "date":{ "parse_upper_inclusive" : false } } } }'
    c. open index.
    curl -XPOST 'localhost:9200/test/_open'
    d. execute query

But I'm sorry, I don't know true role of "index.mapping.date.parse_upper_inclusive" property.

regards,


Jun Ohtani
johtani@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

On 2013/10/04, at 16:00, Jun Ohtani johtani@gmail.com wrote:

Hi,

I am interested this case.
And I try to send range query to _validate api.

curl -XPOST 'localhost:9200/test/logstash/_validate/query?pretty&explain=true' -d '
{
"query_string": {
"query": "@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]"
}
}'

and I recieve strange response.

{
"valid" : true,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"explanations" : [ {
"index" : "test",
"valid" : true,
"explanation" : "@fields.date:[970524000000 TO 23835600999]"
} ]

I use ES 0.90.5.

Does this information helpful?


Jun Ohtani
johtani@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

On 2013/10/04, at 14:46, David Pilato david@pilato.fr wrote:

I was expecting such a trace.
So that's strange the previous form did not work.

I have no more idea yet.

Could you reproduce it in a curl recreation (delete index, create mapping, put doc in between thoses dates, refresh, search with query string)? And Gist it.

It would help to understand. I'm probably missing something.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 07:21, subin ks ksubins321@gmail.com a écrit :

ES errors out with this format. An excerpt from the error stack:

at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.ElasticSearchParseException: failed to parse date field [2013-10-02], tried both date format [MMM dd HH:mm:ss||MMM d HH:mm:ss], and timestamp number
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:194)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:49)
at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:23)
at org.elasticsearch.index.mapper.core.DateFieldMapper.rangeQuery(DateFieldMapper.java:310)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuerySingle(MapperQueryParser.java:377)
at org.apache.lucene.queryparser.classic.MapperQueryParser.getRangeQuery(MapperQueryParser.java:331)
at org.apache.lucene.queryparser.classic.QueryParser.Term(QueryParser.java:441)
at org.apache.lucene.queryparser.classic.QueryParser.Clause(QueryParser.java:257)
at org.apache.lucene.queryparser.classic.QueryParser.Query(QueryParser.java:181)
at org.apache.lucene.queryparser.classic.QueryParser.TopLevelQuery(QueryParser.java:170)
at org.apache.lucene.queryparser.classic.QueryParserBase.parse(QueryParserBase.java:121)
at org.elasticsearch.index.query.QueryStringQueryParser.parse(QueryStringQueryParser.java:214)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.FilteredQueryParser.parse(FilteredQueryParser.java:71)
at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:207)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:255)
at org.elasticsearch.search.query.QueryParseElement.parse(QueryParseElement.java:33)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:549)
... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "2013-10-02"
at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
at org.elasticsearch.common.joda.DateMathParser.parseStringValue(DateMathParser.java:188)
... 30 more


Subin

On Fri, Oct 4, 2013 at 10:42 AM, David Pilato david@pilato.fr wrote:
Try

@fields.date:["2013-10-02" TO "2013-10-03"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 06:28, subin ks ksubins321@gmail.com a écrit :

Hi David,

It doesn't return any result at all, not even the logs which have those dates as time-stamps.

Re,


Subin

On Fri, Oct 4, 2013 at 9:51 AM, David Pilato david@pilato.fr wrote:
Try:

@fields.date:["Oct 2 22:00:00" TO "Oct 3 21:00:00"]

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 4 oct. 2013 à 04:52, subin ksubins321@gmail.com a écrit :

Hello,

I have a Logstash > ES (v 0.90.5) setup. I have date/time of the format - Oct 3 22:30:55. I have this mapping as a template in ES. You may see that @fields.date mapping is set as - "date": { "type": "date", "format": "MMM dd HH:mm:ss||MMM d HH:mm:ss", "index": "analyzed" },

@fields.date is also the field I'll need to execute the date-range queries. Below is the query I execute, on Kibana 0.2.0, to achieve range-searches, but fails:

@fields.date:"Oct 2 22:00:00" TO @fields.date:"Oct 3 21:00:00"

This query returns all logs with the timestamp of Oct 2 22:00:00 and Oct 3 21:00:00, not "between" that time-range.

Please advise on how to achieve the range-query.

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/XeUA1lbd71I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #9