Pick latest data

Elastic Stack Kibana
1

Hi,
I am bulk inserting my data into an index multiple times in a day. I want to get all data with last timestamp(with last timestamp I inserted.).

2

Hi @vikram_singh,

Since you tagged this in kibana-query-language, I'll assume you are referring to queries in the Kibana UI (either in Discover, Visualize, Dashboards, Maps, etc.).

In those apps, you can select the time frame to query via the Time Picker in the UI (mind the box with "Last 24 hours" on the right of the following screenshot):

image
image3360×318 39.5 KB

If you are referring to Elasticsearch queries, via the Search API, you can use the sort property to request for your returned documents to be sorted. As an example:

GET kibana_sample_data_logs/_search
{
  "query": {
    "match_all": {}
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}
3

It will return all data in descending order by timestamp.
But I am searching only all data with only latest time only.

4

Do you mean from a given point in time?

If you know the latest timestamp you got data to act as a from, you can use the Range queries.

Or, if all the entries share the same timestamp, you can run a term query for that exact value.

5

Hi Afharo,
Bulk data is inserting multiple times in a day in index, but not fixed time.
Every time data insert is few thousand docs.
For example data inserting 4 time at different random time point in a day( 1:00 AM, 9:00 AM, 4:00 PM, 8:00 PM).But I don't know which time data is insert.So my query is At 10:00 AM, I want to get only(it is few thousand docs) data inserted at 9:00 AM without knowing last inserted time. At 4:20 PM need to get all data inserted at 4:00 PM.
Only last time inserted all data, without knowing that time point.

6

Hi @vikram_singh,

One last question, when you run your request at 4:20 PM, do you know your last query happened at 10:00 AM?

If so, you can use the Range queries mentioned above. Something like:

GET my_index/_search
{
  "query": {
    "range": {
      "timestamp": {
        "gt": "2020-12-02T10:00:00.000Z",
        "lte": "2020-12-02T16:00:00.000Z"
      }
    }
  }
}

Do you think it might work for your use case?

7

Hi,
I don't know last run query time and last data inserted time.
You suggest range but range will not work in this case.
You can understand top hits of last timestamp data.
Requirement only last inserted bulk data. But query side I don;t know total inserted data at last time.And also don't know what is the last time of data insertion.
The required result is showing in kibana list visualization with top hits on timestamp but by query, not able to get same result.

8

Hi @vikram_singh,

If you got a Kibana visualization working for you, then you can obtain the underlying ES query by using the "Inspect" button in the Visualization view. This feature shows you the underlying data, as well as the ES request and response.

I hope that helps :slight_smile:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.