Use case:
I'm trying to build a realtime logging client that allows the user to search old logs as well as see new logs come in realtime. So far everything mostly functions as expected except when the first queries are made.
To get the first set of "old" logs (the latest 100 logs up to that point) I make a range query like this
{
"sort": [
{
"@timestamp": {
"order": "desc",
"format": "strict_date_optional_time_nanos",
"numeric_type": "date_nanos",
}
},
],
"_source": ["@timestamp", "message"],
"from": from_index, # 0
"size": size, # 100
"query": {
"bool": {
"must": [
...,
"range": {
"@timestamp": {
"lte": starting_time,
}
}
]
},
},
}
For my realtime logging, I poll the endpoint with the above query after modification to the range field (shown below) and update starting_time
with the timestamp of the latest new log.
"range": {
"@timestamp": {
"gt": starting_time,
}
}
In the beginning the starting_time
for both "old" and "new realtime logs" should be the same indicating that it should get all available logs and starting_time
won't ever update in the realtime logging query unless a new log is actually retrieved.
The problem:
On the occasion the above queries are first made (the user enters the page and the first old set of logs + realtime polling logs are retrieved), the realtime polling logs occasionally skip an entry or 2 as show with an example below.
2023-01-17T01:24:52.250619185Z: INFO:root:sleeping for 1 second at 186
2023-01-17T01:24:53.251800193Z: INFO:root:sleeping for 1 second at 187
# Above is the first set of "old logs" before time "2023-01-17T01:24:55.723Z"
------------------------------------------------------------------------------------------
# Below is the realtime logs after time "2023-01-17T01:24:55.723Z"
2023-01-17T01:24:56.255059829Z: INFO:root:sleeping for 1 second at 190
2023-01-17T01:24:57.256236468Z: INFO:root:sleeping for 1 second at 191
I thought it might be a race condition with the insert/retrieve of the logs after the start_time
but I also attempted to wait a few seconds before retrieving the first log after start_time
and the behavior is still the same.
I confirmed that the dates that were missing exist (i refresh the page and they appear in the set of "old logs")