How to do a full update each day

Elastic Stack Elasticsearch
1

Hello,

I use Kibana with elasticsearch. I’m new to Elasticsearch. I have a php rest api with analytic values.

Although this sounds very inefficient, my plan is to query my php rest api to get hourly data year to date, and then each hour completely clear my elasticsearch index of all existing data, and repopulate it with the year to date results of the rest api.

This way, if my elasticsearch fails to pull in the rest api data on one hour, which happens, it will be fixed on the next hour.

How do I need to approach this. Is the idea that I only have one document, with say ID 1, and then I constantly remove and repopulate that document every hour?

Is there a best practice for this sort of thing?

2

Hi,

I have a similar case, I populate a datetime index each 10 minutes to have something like:

my_index_201906061500 then If the index have data and nothing fail I set an alias let's say my_alias.
Then at 15:10 I get again my fresh data and store them in my_index_201906061510 and if no exception and I have data in my index I switch the alias from my_index_201906061500 to my_index_201906061510 and I drop the old index.
I always query on my_alias, this way I always have data in worst case if something go wrong I have the data of 10 minutes ago.

If it fail you send a mail and fix :slight_smile:.

Hope it help!