I want to use elasticsearch only for 1 hour twice a day (I know specific time). The indexing can be done once per day anytime we want.
Is there anyway I can design the system so that it can handle such request and not waste resources for the rest of the day.
I have checked if I can achieve this using scaling of Elasticsearch but it seems that it is not possible to autoscale in Elasticsearch.
Why not use our Elasticsearch Service? You can spin that up for as long or short as you want 
I have already deployed the Elasticsearch on Kubernetes in GCP. How can I modify this.
I dunno sorry, not a k8's person.
I am a novice when it comes to ELK. That being said:
Whether or not you have the service running, you will have all your logstash or beats instances continually pinging your point, awaiting, which wastes bandwidth etc. If you want to only have elasticsearch run for an hour, once or twice a day you should create a cronjob to start the service and then have another cron which will every 5 minutes do something akin to: if (elasticsearch is running){stop service after 2 hours }else {do nothing}
Then what will happen is that whenever you start the instance, within 5 minutes a shutdown timer will start. So then you can just define the cron which will start the job if it isnt already started.
Not sure what you are trying to gain by keeping it off personally. Not a fan of creating windowed systems like what you want. It creates an issue where if there are issues, logstash may never find the system. IE: Logstash has intermittant issues with internet but is never able to access the network or endpoint in the elasticsearch window.
Common use case i think of: Botnets.
nodes may go dark and build backlogs as they could be moved in and out of internet access. Whenever you define the window to send (or send the second internet is available), you need to be able to catch it at all times.
1 Like
@fallenreaper The use case for such availability for specific time is mail service to some people. Search an index and send the results on a mail.
I actually need this for rest of day for bot requests too which are very few compared to what I need for 1 hour. So, please suggest a design for this.
While i dont know kube, the simple solution is to update your crontab and do 1 task for starting the elastic service, and the other for stopping creating a timer. The hardest part is the shell script but the logic would be:
is this script already running? return
is service NOT running? return
start timer for 2 hours
systemctl stop elasticsearch
I shall use circle ci/ travis ci to make elasticsearch cluster on kubernetes.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.