Hello,
I am using Elasticstack to visualise my log data. My log data is iis server log data and after each day a dump for that day is collected from the server at the end of the day and using Logstash the data is indexed into the elastic search. Previously it used to take 2 hours to index a days data and after a while by configuring logstash properly I was able to bring it down to 15 minutes. But the whole setup is being run on a single machine (Ubuntu 16.04), and I am running everything as a service. After indexing the data I am using kibana. But the queries and visualisations I run using Kibana are taking a lot of time, if I just run it on a weeks data. Our plan is to increase it to 6 months. So the lag is giving us troubles. I am using elastic with default configuration.
Daily log data dump is roughly 600-700 MB and after indexing it takes about 1-1.2 GB.
I want to increase it's performance, so can you please suggest the right configuration of elastic (and if possible the machine as well) so that the performance is optimal. Please keep in mind that I want to run Elastic as a service and on a single machine as of now. Do I need to configure my machine more or elastic more? Each query I run takes about 15-20 seconds and aim is to get it under 2-3 seconds. Please ignore if there are any mistakes.
Thanks in advance 