Performance tips wanted for one time logs import

I use an ELK stack to process sets of historical logs which I receive from customers in zipped up bundle.

In total there could be 2~3 million lines of logs which I want to import quickly so I can start analysing them in Kibana.

However these logs can take a long time to be fully processed by elasticsearch / kibana. I can see my index growing slowly and it could be 4 - 5 hours before everything is imported.

After I'm done with them I will delete them from elasticsearch, and later on import a new set.

Can anyone give me some configuration options that will help speed up the import of these logs?

I have 4 cores in my system and I notice my cpu is averaging 30-40% utilization during the import.

Is it logstash or elastic that does most of the heavy lifting? I want my CPU's to be working much harder!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.