Keep older data in kibana when using a python script to upload data

Elastic Stack Kibana
1

Hello there !

Sorry if my question seems stupid, it is my first time touching ELK and also JSON files, I'm doing an internship related to those topics and I need a bit of help.

So I have a python script that open a json file with various information and then send it to elasticsearch. From there, I can connect directly with my navigator with localhost:5601, manage the index and add the data on a dashboard.

Here is my code, uploading the information :

import requests
import json

res = requests.get('http://localhost:9200')

file = open("data_cores.json", "r")
line=file.readline()
tab = []

#read the line in the json file 
#and delete the comma at the end, then ad it in tab
while line:
    tab += [line]
    line = file.readline()
i=0
while i<5:
    tab[i] = tab[i][:-2]
    i=i+1

#I changed the name of my variables 
item1 = tab[0]
item2= tab[1]
item3= tab[2]
item4= tab[3]


from elasticsearch import Elasticsearch
es = Elasticsearch(
    ['localhost'],
    port=9200
)

res = es.index(index="indexproject", id=1, body=item1)
res = es.index(index="indexproject", id=2, body=item2)
res = es.index(index="indexproject", id=3, body=item3)
res = es.index(index="indexproject", id=4, body=item4)

and the structure of my json file :

{"name": "item 1", "cores": 0, "date" : },
{"name": "item 2", "cores": 44},
{"name": "item 3", "cores": 726},
{"name": "Item 4", "cores": 9}

The problem is, I want that everytime I run the python script, the data is uploaded in elasticsearch and the dashboard is updated but with a "time mark" somewhere, in such a way that it doesn't erase the older data and I can change the date on the dashboard to see previous days.
I'm pretty sure that I have to change something in the structure of the documents I send to elasticsearch but I'm totally lost.

Thank you in advance.

EDIT : I have add a date on my json document, Kibana can read the date, no problem. But when I launch again the script, the new data still erase the older ones, even if the date is not the same

2

If the id value already exists, it will overwrite the elasticsearch document. There's a few options here - if there's a unique id that you can use, that will work. If there's no need to edit these afterwords, not specifying an id will cause elasticsearch to generate one for you.

1 Like
3

Thank you very much, it's working now ! :grin: I just deleted the "id=XX" in the es.index and it's all good !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.