I use an ELK stack to process sets of historical logs which I receive from customers in zipped up bundle.
In total there could be 2~3 million lines of logs which I want to import quickly so I can start analysing them in Kibana.
However these logs can take a long time to be fully processed by elasticsearch / kibana. I can see my index growing slowly and it could be 4 - 5 hours before everything is imported.
After I'm done with them I will delete them from elasticsearch, and later on import a new set.
Can anyone give me some configuration options that will help speed up the import of these logs?
I have 4 cores in my system and I notice my cpu is averaging 30-40% utilization during the import.
Is it logstash or elastic that does most of the heavy lifting? I want my CPU's to be working much harder!