I found an issue that may be related to this. About a week ago in response to this issue, I changed my template mapping for timestamps to use epoch_second
. Logs were previously stored like this:
"@timestamp": "2020-02-03 23:59:00"
And now they're stored like this:
"@timestamp": 1582070340
I just realized that since making that change, none of those logs are being returned in the queries being made by the visualizations. The reason is because they filter for timestamps using epoch_millis
, like this:
{
"aggregations": {
"top_countries_count": {
"cardinality": {
"field": "source.geo.country_iso_code"
}
},
"source": {
"terms": {
"field": "source.geo.country_iso_code",
"size": 10,
"order": {
"bytes_out": "desc"
}
},
"aggs": {
"bytes_in": {
"sum": {
"field": "destination.bytes"
}
},
"bytes_out": {
"sum": {
"field": "source.bytes"
}
},
"flows": {
"cardinality": {
"field": "network.community_id"
}
},
"source_ips": {
"cardinality": {
"field": "source.ip"
}
},
"destination_ips": {
"cardinality": {
"field": "destination.ip"
}
}
}
}
},
"query": {
"bool": {
"filter": [
{
"bool": {
"must": [],
"filter": [],
"should": [],
"must_not": []
}
},
{
"range": {
"@timestamp": {
"gte": 1582070330,
"lte": 11582070350
}
}
}
]
}
}
}
Again, notice this part:
"range": {
"@timestamp": {
"gte": 1582070330,
"lte": 11582070350
}
}
That returns nothing. When I switch it to instead use epoch_seconds
, it returns my logs.
I'm still not sure if that's related to this issue - you think I should open another ticket for that?