Indexing and range querying with _timestamp (is there an example?)

Elastic Stack Elasticsearch
1

Hi all,
I'm struggling to understand _timestamp and how to use it correctly. I'm
storing log data so the timestamp is very important. I'd like to be able to
store the timestamp and query it by range. Whenever I a field with the name
_timestamp I can never seem to get any results when I query by it. This
happens whether I store and query using ISO 8601 text format or with millis
as longs. Does anybody have a complete example of this working?

Test code when using text format (FWIW, this is unit test code running as a
local node).

client.admin().indices().create(new
CreateIndexRequest(indexName).mapping("log",
"{"_timestamp": {"enabled": true, "store":
"yes"}}"));
client.prepareIndex(indexName, "log", auditEvent.getId().toString())
.setSource(jsonBuilder()
.startObject().
field("eventType",
auditEvent.getEventType().toString()).
field("_timestamp",
auditEvent.getEventTime().toString(ISODateTimeFormat.dateTime())).
field("userId", auditEvent.getUserId()).
endObject())
.execute()
.actionGet();

SearchResponse response = client.prepareSearch(indexNames.toArray(new
String[] {})).setTypes("log")
.setQuery(
QueryBuilders.
boolQuery()
.must(QueryBuilders.fieldQuery("eventType",
eventType))
.must(QueryBuilders.rangeQuery("_timestamp")
.from(interval.getStart().toString())
.to(interval.getEnd().toString())
)
)
.addField("_timestamp")
.execute().actionGet();

When I do the same query with a match all I can see the timestamp fields.
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "3d584830-8506-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T20:01:02.003Z"
}
}, {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "e6e382e0-84f5-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T18:04:05.006Z"
}
} ]

What am I not understanding?

Many thank, in advance, for your help.

Cheers,
Edward

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Try giving it a range that's larger than expected, and see if you get any results. The docs say it is stored as a long. But, I am not sure if the precision is maintained during searches. At least so far with a few tests, I have found that milliseconds precision is NOT maintained.

On Mar 19, 2013, at 5:19 PM, Edward Sargisson ejsarge@gmail.com wrote:

Hi all,
I'm struggling to understand _timestamp and how to use it correctly. I'm storing log data so the timestamp is very important. I'd like to be able to store the timestamp and query it by range. Whenever I a field with the name _timestamp I can never seem to get any results when I query by it. This happens whether I store and query using ISO 8601 text format or with millis as longs. Does anybody have a complete example of this working?

Test code when using text format (FWIW, this is unit test code running as a local node).

client.admin().indices().create(new CreateIndexRequest(indexName).mapping("log",
"{"_timestamp": {"enabled": true, "store": "yes"}}"));
client.prepareIndex(indexName, "log", auditEvent.getId().toString())
.setSource(jsonBuilder()
.startObject().
field("eventType", auditEvent.getEventType().toString()).
field("_timestamp", auditEvent.getEventTime().toString(ISODateTimeFormat.dateTime())).
field("userId", auditEvent.getUserId()).
endObject())
.execute()
.actionGet();

SearchResponse response = client.prepareSearch(indexNames.toArray(new String {})).setTypes("log")
.setQuery(
QueryBuilders.
boolQuery()
.must(QueryBuilders.fieldQuery("eventType", eventType))
.must(QueryBuilders.rangeQuery("_timestamp")
.from(interval.getStart().toString())
.to(interval.getEnd().toString())
)
)
.addField("_timestamp")
.execute().actionGet();

When I do the same query with a match all I can see the timestamp fields.
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "3d584830-8506-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T20:01:02.003Z"
}
}, {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "e6e382e0-84f5-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T18:04:05.006Z"
}
} ]

What am I not understanding?

Many thank, in advance, for your help.

Cheers,
Edward

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Vinh.

The answer ended up being that the timestamp needs to be set on the
IndexRequest with the timestamp method. Merely adding it as a field called
_timestamp does not work.

On Wed, Mar 20, 2013 at 11:05 AM, vinh vinh@loggly.com wrote:

Try giving it a range that's larger than expected, and see if you get any
results. The docs say it is stored as a long. But, I am not sure if the
precision is maintained during searches. At least so far with a few tests,
I have found that milliseconds precision is NOT maintained.

On Mar 19, 2013, at 5:19 PM, Edward Sargisson ejsarge@gmail.com wrote:

Hi all,
I'm struggling to understand _timestamp and how to use it correctly. I'm
storing log data so the timestamp is very important. I'd like to be able to
store the timestamp and query it by range. Whenever I a field with the name
_timestamp I can never seem to get any results when I query by it. This
happens whether I store and query using ISO 8601 text format or with millis
as longs. Does anybody have a complete example of this working?

Test code when using text format (FWIW, this is unit test code running as
a local node).

client.admin().indices().create(new
CreateIndexRequest(indexName).mapping("log",
"{"_timestamp": {"enabled": true, "store":
"yes"}}"));
client.prepareIndex(indexName, "log", auditEvent.getId().toString())
.setSource(jsonBuilder()
.startObject().
field("eventType",
auditEvent.getEventType().toString()).
field("_timestamp",
auditEvent.getEventTime().toString(ISODateTimeFormat.dateTime())).
field("userId", auditEvent.getUserId()).
endObject())
.execute()
.actionGet();

SearchResponse response = client.prepareSearch(indexNames.toArray(new
String {})).setTypes("log")
.setQuery(
QueryBuilders.
boolQuery()
.must(QueryBuilders.fieldQuery("eventType",
eventType))
.must(QueryBuilders.rangeQuery("_timestamp")
.from(interval.getStart().toString())
.to(interval.getEnd().toString())
)
)
.addField("_timestamp")
.execute().actionGet();

When I do the same query with a match all I can see the timestamp fields.
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "3d584830-8506-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T20:01:02.003Z"
}
}, {
"_index" : "audit-events-2013-03-04",
"_type" : "log",
"_id" : "e6e382e0-84f5-11e2-8365-24be05270b5c",
"_score" : 1.0,
"fields" : {
"_timestamp" : "2013-03-04T18:04:05.006Z"
}
} ]

What am I not understanding?

Many thank, in advance, for your help.

Cheers,
Edward

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/Oo9KfK0E7P4/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

On Wed, 2013-03-20 at 11:05 -0700, vinh wrote:

Try giving it a range that's larger than expected, and see if you get
any results. The docs say it is stored as a long. But, I am not sure
if the precision is maintained during searches. At least so far with
a few tests, I have found that milliseconds precision is NOT
maintained.

That's not correct - millisecond precision is maintained.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hmm…perhaps it is because I'm adding it as a field named "_timestamp" and setting the value as a long? Gonna try Edward's approach and pass it directly to IndexRequest.setTimestamp(). Though I will need to first convert the long to string since the method requires a string.

Side note…would definite be good if it took a long, since ES will eventually convert to and store as a long anyways.

On Mar 21, 2013, at 4:31 AM, Clinton Gormley clint@traveljury.com wrote:

On Wed, 2013-03-20 at 11:05 -0700, vinh wrote:

Try giving it a range that's larger than expected, and see if you get
any results. The docs say it is stored as a long. But, I am not sure
if the precision is maintained during searches. At least so far with
a few tests, I have found that milliseconds precision is NOT
maintained.

That's not correct - millisecond precision is maintained.

clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.