Elasticsearch get results only appeared in last 30 seconds

[yolo_1900]

I have two Python client code. One runs each 30 seconds and submits data to elasticsearch. The other one runs each 30 seconds, download the data that is submitted by the first program and analyze the data.

In the second program, I want to limit the SEARCH function to get the data that is submitted in last 30 seconds only (because the earlier data is already downloaded).

I have a search command (modified from https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html):

POST /my_index_*/_search
{
  "size": 10,
  "query": {
    "match_all": {},
    "range": {
      "date": {
        "gte": "now-30s/d",
        "lt": "now/d"
      }
    }
  }
}

But it returns an error.

{
  "error": {
    "root_cause": [
      {
        "type": "parsing_exception",
        "reason": "[match_all] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
        "line": 5,
        "col": 5
      }
    ],
    "type": "parsing_exception",
    "reason": "[match_all] malformed query, expected [END_OBJECT] but found [FIELD_NAME]",
    "line": 5,
    "col": 5
  },
  "status": 400
}

I think certainly the error comes from the range condition because without that I can get the data well.

How should I do that? And is there a better way than limit the duration of 30 seconds to make sure that the second program never has a data more than once, but also do not miss any data.

Many thanks

[mikebd (Michael Ben-David)]

It seems this may be a dangerous approach, susceptible to both duplicates and misses (more likely). This does what I think you are looking for:

GET /my_index_*/_search
{
  "size": 10,
  "query": {
    "bool": {
      "filter": {
        "range": {
          "date": {
            "gte": "now-30s",
            "lt": "now"
          }
        }
      }
    }
  }
}