@timestamp range - searching across indices

Elastic Stack Elasticsearch
1

Hi all,

I am new to Elasticsearch and would like to know if it's possible to retrieve all logs over the last 24 hours while searching across all indices?

I am running the below queries using Sense. When I run the first query it selects data as expected but when I run the second query it selects dates from days ago

Query:
GET /logstash-sdnet-2016.06.08,logstash-sdnet-2016.06.09/_search
{
"filter":{
"range":{"@timestamp":{"gte":"now-24h"}}}
}

Results:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 19431,
"max_score": 1,
"hits": [
{
"_index": "logstash-sdnet-2016.06.08",
"_type": "collectd",
"_id": "AVTcO26tvGUlWg6s43sU",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-06-08T08:06:45.000Z",
"plugin": "UPS",
"value": 0,
"type": "collectd",
"metric": "apc_smartups_BattTimeOn",
"customer": "SDNet"
}
},
{
"_index": "logstash-sdnet-2016.06.08",
"_type": "collectd",
"_id": "AVTcO26tvGUlWg6s43sa",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-06-08T08:06:45.000Z",
"plugin": "UPS",
"value": 247.6,
"type": "collectd",
"metric": "apc_smartups_OutputV",
"customer": "SDNet"
}
},

Query:
GET /logstash-sdnet-*/_search
{
"filter":{
"range" : {
"@timestamp" : {
"gt" : "now-24h"}}}
}

Results:
{
"took": 54,
"timed_out": false,
"_shards": {
"total": 650,
"successful": 650,
"failed": 0
},
"hits": {
"total": 173882,
"max_score": 1,
"hits": [
{
"_index": "logstash-sdnet-2016.05.23",
"_type": "collectd",
"_id": "AVS0qT1MvGUlWg6s4R-C",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-05-23T05:16:45.000Z",
"plugin": "UPS",
"value": 100,
"type": "collectd",
"metric": "apc_smartups_BattCapacity",
"customer": "SDNet"
}
},
{
"_index": "logstash-sdnet-2016.05.23",
"_type": "collectd",
"_id": "AVS0qT1MvGUlWg6s4R-E",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-05-23T05:16:45.000Z",
"plugin": "UPS",
"value": 246.2,
"type": "collectd",
"metric": "apc_smartups_InputVmax",
"customer": "SDNet"
}
},

Mapping:
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},

Do you always need to specify the index when searching for date ranges?

Many thanks!

2

No, the second query should also have worked (only returned hits in the past day).

The query looks correct to me, but to be sure, can you try specifying some older index names (that have no hits within the past day) and confirm you get 0 hits?

Mike McCandless

3

Hi there,

If I run the below query for 2016.05.25 and 2016.05.26 I get hits

GET /logstash-sdnet-2016.05.25,logstash-sdnet-2016.05.26/_search
{
"filter":{
"range":{"@timestamp":{"gte":"now-24h"}}}
}

{
"took": 2,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 31437,
"max_score": 1,
"hits": [
{
"_index": "logstash-sdnet-2016.05.25",
"_type": "collectd",
"_id": "AVS8EgH2vGUlWg6s4ZCA",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-05-25T02:36:45.000Z",
"plugin": "UPS",
"value": 0,
"type": "collectd",
"metric": "apc_smartups_BattTimeOn",
"customer": "SDNet"
}
},
{
"_index": "logstash-sdnet-2016.05.25",
"_type": "collectd",
"_id": "AVS8EgH2vGUlWg6s4ZCF",
"_score": 1,
"_source": {
"host": "ups.l.dwyer.id.au",
"@timestamp": "2016-05-25T02:36:45.000Z",
"plugin": "UPS",
"value": 50,
"type": "collectd",
"metric": "apc_smartups_InputFreq",
"customer": "SDNet"
}
},

:frowning:

4

Hmm, maybe try the explain API?

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-explain.html

5

Thank you!

I'll give this a try and see what I find