Keep older data in kibana when using a python script to upload data

Hello there !

Sorry if my question seems stupid, it is my first time touching ELK and also JSON files, I'm doing an internship related to those topics and I need a bit of help.

So I have a python script that open a json file with various information and then send it to elasticsearch. From there, I can connect directly with my navigator with localhost:5601, manage the index and add the data on a dashboard.

Here is my code, uploading the information :

import requests
import json

res = requests.get('http://localhost:9200')

file = open("data_cores.json", "r")
line=file.readline()
tab = []

#read the line in the json file 
#and delete the comma at the end, then ad it in tab
while line:
    tab += [line]
    line = file.readline()
i=0
while i<5:
    tab[i] = tab[i][:-2]
    i=i+1

#I changed the name of my variables 
item1 = tab[0]
item2= tab[1]
item3= tab[2]
item4= tab[3]


from elasticsearch import Elasticsearch
es = Elasticsearch(
    ['localhost'],
    port=9200
)

res = es.index(index="indexproject", id=1, body=item1)
res = es.index(index="indexproject", id=2, body=item2)
res = es.index(index="indexproject", id=3, body=item3)
res = es.index(index="indexproject", id=4, body=item4)

and the structure of my json file :

{"name": "item 1", "cores": 0, "date" : },
{"name": "item 2", "cores": 44},
{"name": "item 3", "cores": 726},
{"name": "Item 4", "cores": 9}

The problem is, I want that everytime I run the python script, the data is uploaded in elasticsearch and the dashboard is updated but with a "time mark" somewhere, in such a way that it doesn't erase the older data and I can change the date on the dashboard to see previous days.
I'm pretty sure that I have to change something in the structure of the documents I send to elasticsearch but I'm totally lost.

Thank you in advance.

EDIT : I have add a date on my json document, Kibana can read the date, no problem. But when I launch again the script, the new data still erase the older ones, even if the date is not the same

If the id value already exists, it will overwrite the elasticsearch document. There's a few options here - if there's a unique id that you can use, that will work. If there's no need to edit these afterwords, not specifying an id will cause elasticsearch to generate one for you.

1 Like

Thank you very much, it's working now ! :grin: I just deleted the "id=XX" in the es.index and it's all good !