Point in Time with Search After missing a few logs

Elastic Stack Elasticsearch
1

When I run a regular query search in elasticsearch, I will get all the values. Now, when I implement the point in time api and search after because using the regular search api can handle up to 10K hits/results the query skips a few logs (3).

The following cmd has 88 logs. It will only grab 85 of them. I had one user who had 91 logs and when I ran the same command for the PIT and search after, all the logs were recorded. I have no idea why this is skipping it, for 88 logs but not the user that has 91 logs. Is there something wrong with my PIT and search after cli cmd?

In elasticsearch CLI, when I run the cmd:

GET /app-logs-*/_search
{
  "size": 100,
  "_source": ["message"], 
  "query": {
    "bool": {
      "should": [
            {
              "match": {
                "fields.User": "mike"
              }
            }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
                "gte": "2021-08-01T00:00:00.000-05:00",
                "lt": "2021-09-05T23:59:59.999-05:00"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  },
  "sort": [
   {
      "@timestamp": {
        "order": "asc"
      }
    }
  ]
}

but when I run the cmd with search after and PIT:

POST /app-logs-*/_pit?keep_alive=1m

GET /_search
{
  "track_total_hits": false,
  "size": 20,
  "_source": ["message"], 
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "fields.User": "mike"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
               "gte": "2021-06-01T00:00:00.000-05:00",
                "lt": "2021-09-05T23:59:59.999-05:00"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  },
  "pit": {
    "id": "85ezAwMzY3VzdG9tZXItc2ltdWxhdGlvbi1lcy1hcHAtbG9ncy1kZXZlbG9wbWVudC0yMDIxLTA3FlNMZWlBd1lRUXpDaGhLMXJHbEtKVHcAFmhaQ3NlRDVKUjBHdjI0dGhrblJpTWcAAAAAAAAAJf8WLTVKdUd1cm9SLVNFNVozVEJOY2xZUQAzY3VzdG9tZXItc2ltdWxhdGlvbi1lcy1hcHAtbG9nyenaais8G9wbWVudC0yMDIxLTA5FmxyLXhJZkNkUjgyblhfUUI0c1ViWXcAFmhaQ3NlRDVKUjBHdjI0dGhrblJpTWcAAAAAAAAAJgEWLTVKdUd1cm9SLVNFNVozVEJOY2xZUQAzY3VzdG9tZXItc2ltdWxhdGlvbi1lcy1hcHAtbG9ncy1kZXZlbG9wbWVudC0yMDIxLTA4FlRNT0IwQndlUUZla0FTYVk5ejFDS1EAFmhaQ3NlRDVKUjBHdjI0dGhrblJpTWcAAAAAAAAAJgAWLTVKdUd1cm9SLVNFNVozVEJOY2xZUQADFlRNT0IwQndlUUZla0FTYVk5ejFDS1EAABZTTGVpQXdZUVF6Q2hoSzFyR2xLSlR3AAAWbHIteElmQ2RSODJuWF9RQjRzVWJZdwAA",
    "keep_alive": "1m"
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ],
  "search_after": [
    1630682235382
    ]
}

the 40th log will get read, the 41th log should have been next, but it is skipped and the 42th log is read.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.