I've been using Elasticsearch for a few weeks, and now I also installed Kibana to create some visualizations for my data.
I have a Python script which I open in three instances (I collect data with three different sensors, and each of them uses its own folder to save its logs) and which uses the elasticsearch-py library to add data to my local database by reading some log files from the log folders I mentioned, every few seconds. Everything works just well... until I close my scripts.
After closing the scripts, no new data should be added to the database. And now the funny part begins, because Elasticsearch keeps adding documents from all three devices, with new ids and the values from the last working POST (meaning, if the sensor reading from device no. 2 before closing was 23° C, then all subsequent entries for device no. 2 will use the same temperature - even if said device is shut down and my script closed).
I have absolutely no idea what is causing this; I've tried restarting Elasticsearch and Kibana, restarting the PC, disconnecting the sensors, even deleting the index and populating it again; it stays empty until I start adding documents, then it continues adding them the way I mentioned even after stopping everything else.
I've attached a screenshot from Kibana with the last entries; as you can see, there are entries from today at 16:04 - too bad that I shut down the sensors and every script that could make REST calls at 15:43.
I've searched everywhere but found nothing; I wouldn't be posting here if I had any idea about what could be the problem.
Can your problem come from a timezone configuration, maybe the data that you collect at 15:00 are stored in elastic with a timestamp of 15:00 but in kibana it will show at 19:00 as your timezone is not UTC... it's just one guess.
Another idea:
If you make a count of your document just after you stop your service and a count 1 hour after and compare if there's more document. It make no sense for elatic to add document by it self, it certainly come from somewhere maybe your script are not killed correctly.
One last quetion but did you check about filebeat to send your logs to elastic??
Yes, I definitely have the timezone problem. The data I insert in the database (which has the date and time of the device that has saved it, namely, the current date and time, since I'm collecting data in almost real-time) is shown in Kibana with a delay of two hours; I've been fiddling with the timezone setting, but couldn't find a definitive solution. I'm currently reinstalling both Kibana and Elasticsearch, and I will try to repopulate the db after setting the timezone (not before), and see if this will solve the problem. It is possible that my problem originates from the timezone, as you suggested, since everything else seems to work correctly.
The scripts I use are killed correctly, I've doubled checked that before opening this thread. The docs count increases anyway.
I didn't even know about Filebeat. The project I'm working on is for my thesis, so it's better if I make my own script to send data to the server instead of using another pre-made tool. Anyway, it looks pretty neat; I'll make sure to remember it in the future, thanks for the suggestion.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
