How to retrieve only latest index

I have created a python api which will accept the *.json file.

for ex: abc.json

{
    "service_name": "httpd",
    "service_status": "DOWN",
    "host_name": "host1",
    "time": "1616600149.014236"
}

and, push the data to the ES using below python api.

@app.route('/add', methods=['POST']) def insert_data():
    #directory = '/home/user'
    dir = os.getcwd()
    os.chdir(dir)
    i = 1
    #This function will read all the json payload in the given dir and upload to ES.
    for filename in os.listdir(dir):
        if filename.endswith(".json"):
            f = open(filename)
            status_content = f.read()
            # Send the data into es
            result=(es.index(index='svc_index', ignore=400, doc_type='doc',
            id=i, body=json.loads(status_content)))
            i = i + 1
            print("result")

    return jsonify(result)

Output:

{
        "_id": "4", 
        "_index": "svc_index", 
        "_score": 1.0, 
        "_source": {
          "host_name": "host1", 
          "service_name": "httpd", 
          "service_status": "DOWN", 
          "time": "1616600143.5427265"
        }, 
        "_type": "doc"
      },

Since timestamp is being stored as string, sorting is not working. I wanted to bring the latest result on top. Could anyone please help into this.

Thanks!!

Welcome to our community! :smiley:

Which one is the timestamp?

Thanks Warkolm.

I have updated the time field to store in json file only in time format not in str.
"time": 1616600143.5427265

and, still with the following code, when I am retrieving the latest result with desc order@time. it's not working.

result = es.search(index="svc_index", doc_type="doc", sort='time:desc', body={"query": {"match": {"service_name": query}}}, size=1)

Thanks!!

Ideally you would translate that to a proper time format so that you can sort on it. Is that possible?

see , the timestamp of this file

[azureuser@harpr-mac]$ cat httpd-status-1616618301.5598042.json
{
"service_name": "httpd",
"service_status": "DOWN",
"host_name": "host1",
"time": 1616618301.5458138
}

and with the following es result.

azureuser@harpr-mac]$ curl -X GET http://127.0.0.1:5000/healthcheck/httpd
{
  "_shards": {
    "failed": 0, 
    "skipped": 0, 
    "successful": 1, 
    "total": 1
  }, 
  "hits": {
    "hits": [
      {
        "_id": "4", 
        "_index": "svc_index", 
        "_score": null, 
        "_source": {
          "host_name": "host1", 
          "service_name": "httpd", 
          "service_status": "DOWN", 
          "time": 1616618210.7548866
        }, 
        "_type": "doc", 
        "sort": [
          1616618240.0
        ]
      }
    ], 
    "max_score": null, 
    "total": {
      "relation": "eq", 
      "value": 5
    }
  }, 
  "timed_out": false, 
  "took": 1
}

That is not a valid timestamp for Elasticsearch, it's just a number. You will want it to be something like Date field type | Elasticsearch Reference [7.12] | Elastic for Elasticsearch to be able to translate that into UTC and then sort on.

Agreed, its treating it as number, but still the latest file has greater timestamp 1616618301 than es result 1616618210.

If I use datetime in python to store the value in json, it give serializable error. I need to convert it to string first then in ES mapping which is not working through python script.

I have found the solution.

Use the following sort function with mode=max.

result = es.search(index="svc_index", doc_type="doc", body={"query": {"match": {"service_name": query}},"sort":{"time": {'order': 'desc', 'mode':'max'}}}, size=1)

Thanks for your help!