How to retrieve only latest index

Elastic Stack Elasticsearch
1

I have created a python api which will accept the *.json file.

for ex: abc.json

{
    "service_name": "httpd",
    "service_status": "DOWN",
    "host_name": "host1",
    "time": "1616600149.014236"
}

and, push the data to the ES using below python api.

@app.route('/add', methods=['POST']) def insert_data():
    #directory = '/home/user'
    dir = os.getcwd()
    os.chdir(dir)
    i = 1
    #This function will read all the json payload in the given dir and upload to ES.
    for filename in os.listdir(dir):
        if filename.endswith(".json"):
            f = open(filename)
            status_content = f.read()
            # Send the data into es
            result=(es.index(index='svc_index', ignore=400, doc_type='doc',
            id=i, body=json.loads(status_content)))
            i = i + 1
            print("result")

    return jsonify(result)

Output:

{
        "_id": "4", 
        "_index": "svc_index", 
        "_score": 1.0, 
        "_source": {
          "host_name": "host1", 
          "service_name": "httpd", 
          "service_status": "DOWN", 
          "time": "1616600143.5427265"
        }, 
        "_type": "doc"
      },

Since timestamp is being stored as string, sorting is not working. I wanted to bring the latest result on top. Could anyone please help into this.

Thanks!!

2

Welcome to our community! :smiley:

Which one is the timestamp?

3

Thanks Warkolm.

I have updated the time field to store in json file only in time format not in str.
"time": 1616600143.5427265

and, still with the following code, when I am retrieving the latest result with desc order@time. it's not working.

result = es.search(index="svc_index", doc_type="doc", sort='time:desc', body={"query": {"match": {"service_name": query}}}, size=1)

Thanks!!

4

Ideally you would translate that to a proper time format so that you can sort on it. Is that possible?

5

see , the timestamp of this file

[azureuser@harpr-mac]$ cat httpd-status-1616618301.5598042.json
{
"service_name": "httpd",
"service_status": "DOWN",
"host_name": "host1",
"time": 1616618301.5458138
}

and with the following es result.

azureuser@harpr-mac]$ curl -X GET http://127.0.0.1:5000/healthcheck/httpd
{
  "_shards": {
    "failed": 0, 
    "skipped": 0, 
    "successful": 1, 
    "total": 1
  }, 
  "hits": {
    "hits": [
      {
        "_id": "4", 
        "_index": "svc_index", 
        "_score": null, 
        "_source": {
          "host_name": "host1", 
          "service_name": "httpd", 
          "service_status": "DOWN", 
          "time": 1616618210.7548866
        }, 
        "_type": "doc", 
        "sort": [
          1616618240.0
        ]
      }
    ], 
    "max_score": null, 
    "total": {
      "relation": "eq", 
      "value": 5
    }
  }, 
  "timed_out": false, 
  "took": 1
}
6

That is not a valid timestamp for Elasticsearch, it's just a number. You will want it to be something like Date field type | Elasticsearch Reference [7.12] | Elastic for Elasticsearch to be able to translate that into UTC and then sort on.

7

Agreed, its treating it as number, but still the latest file has greater timestamp 1616618301 than es result 1616618210.

If I use datetime in python to store the value in json, it give serializable error. I need to convert it to string first then in ES mapping which is not working through python script.

8

I have found the solution.

Use the following sort function with mode=max.

result = es.search(index="svc_index", doc_type="doc", body={"query": {"match": {"service_name": query}},"sort":{"time": {'order': 'desc', 'mode':'max'}}}, size=1)

Thanks for your help!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.