Search API error Elasticsearch

[Jose_Angel_Morena_Si (jmorena)]

Hello there,

When I use python elasticsearch module to query for documents, there seems to be something strange going on:

elasticsearch module: version 7.11.2
elasticsearch cluster: version 7.11.2

The following query:

result = es.search(body={},                                                                                                                                                                          index="winlogbeat-*,                       
request_timeout=30
) 

This code only returns documents of the day 12/05, there is no document younger nor older, according to the results of the query.

The problem is that there are of course documents of other days, which can be seen in Kibana.

How can this be happening? What can I be missing?

Thanks!

Kind regards,

[Christian_Dahlqvist]

By default only 10 documents are returned, which I assume could all be from the most recent index. To get more data you need to specify the size, which can not be larger than 10000.

[dadoonet (David Pilato)]

Yes. Some additional information.

You can use:

• the size and from parameters to display by default up to 10000 records to your users. If you want to change this limit, you can change index.max_result_window setting but be aware of the consequences (ie memory).
• the search after feature to do deep pagination.
• the Scroll API if you want to extract a resultset to be consumed by another tool later.

[Jose_Angel_Morena_Si (jmorena)]

Hello there,

I have tried using the from and size fields and there was no effect. The issue is still there.

Apparently there should be more recent docs as I can see in Kibana:

Fyi, there is an alias named winlogbeat-7.11.2 , and an index named winlogbeat-7.11.2-2021.05.09-000002.

Why do you think the SEARCH Api is only returning the docs of May 12th?

Thanks!

Kind regards,

[Christian_Dahlqvist]

Is that alias the write alias? If so it only points to the latest underlying index. Try searching winlogbeat-7.11.2-* and see if that makes a difference.

[Jose_Angel_Morena_Si (jmorena)]

Hello,

I have tried to query directly on the index and also directly on winlogbeat-7.11.2-*. The results are the same:

With or without the from and size parameters, the issue still persists:

The returned docs are still from the 12th of May.

Is there any alternative way of getting the last recent docs of winlogbeat, without using the SEARCH Api?

Thank you!

[Christian_Dahlqvist]

It looks like you have several thousand documents per day and are only returning 2000. Elasticsearch natural ordering may mean you do not get a random sample. I would recommend adding a date range filter to limit the serach or sort descending on the @timestamp field.

[Jose_Angel_Morena_Si (jmorena)]

Hi Christian,

Thanks for your help!

I tried using date range filter and Elasticsearch returned the following:

JSON Response:

{
    "took": 1,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 0,
            "relation": "eq"
        },
        "max_score": null,
        "hits": []
    }
}

Why hits field is empty? I can visualize the docs of today in Kibana.

Kind regards,

[Christian_Dahlqvist]

Do the same search against winlogbeat-* as thst is what you are using in Kibana. Does that return results? Do you have winlogbeat indices for multiple different versions?

[Jose_Angel_Morena_Si (jmorena)]

Hi,

I used the same search against winlogbeat-*, and the results are still the same:

I only have one winlogbeat index, the winlogbeat instances are the same version as elastic stack.

Kind regards,

[Christian_Dahlqvist]

Why are you filtering on the timestamp field and not @timestamp? Does this even exist?

[Jose_Angel_Morena_Si (jmorena)]

Sorry I based the query on the following documentation, in which the examples quote timestamp, Range query | Elasticsearch Guide [7.11] | Elastic

But yes, when using @timestamp instead of timestamp (which makes total sense), it works!

Now I can see the documents of the 30th of May.

Thank you very much!