# ELK Limits, config tips, high CPU usage, cluster (newbie) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 21, 2018, 10:52pm UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410 "2018-01-21T22:52:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sylvain-69](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain-69/32/26561_2.png) [@Sylvain-69](https://discuss.elastic.co/u/Sylvain-69)\ **Post date:** [January 21, 2018, 10:52pm UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/1 "2018-01-21T22:52:22Z")
Hello Everybody ! I started to use ELK in my company, at first to be able to get a look at windows logon informations then members of security groups activity, etc... I installed one node on Centos 7 (VM) with 2 CPUs (Virtual sockets) and 8 Go RAM. ELK was installed with the default configuration, I only set the basics to get it work. Of course after digging a little bit, I found out ELK to be very usefull, so I started to send more logs into it (netflow, winlogbeat), and now we are asked to monitor network activity on some PCs (so I used Packetbeat), etc... I did manage for winlogbeat and packbeat to send to ELK only the necessary (only the events needed and only the traffic from/to the IPs needed) , but recently I added 2 more netflow logs and now my node is at 100% CPU usage, it seems that it is to much for it... Since it's in "production" and that it will grow because I want to developp the use of ELK in my company. I need advices on my configuration and what to do in the futur (cluster, how many nodes ? do I need Redis ? etc...). I need to know the best practice for the use of netflow codec, csv, geoip, etc and other filters, If there is some tools to monitor logstash I/O, and if you have some tips and how you guys work with ELK to get it working fast and efficient (or any other tips that is usefull), if there is some kind of limits to logstash or elasticsearch ? I know that there is a lot of informations on internet and that google is my best friend, but I already spent a lot of time starting from zero to get where I am, and sometimes it's hard to understand everything I read... So maybe if we can discuss about that it will help me understand better and found the informations I need to evolve ! At the same time if you have some links that can be good for a newbie to read, it will be appreciated ! thanks ! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 4:26am UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/8 "2018-01-22T04:26:43Z")
Bonjour 😉 Please format your code using `` icon as explained in [this guide](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will make your post more readable. Or use markdown style like: ```` ``` CODE ``` ```` For big part of code, you can use [gist.github.com](http://gist.github.com) to share it. Would be nice if you edit your post (move main stuff to gist and delete the part of the post which are not needed anymore). Also FYI there is a french section at #in-your-native-tongue:discussions-en-francais but feel free to continue posting here in english. Just please use English (some comments are in french). 🙂 ---
**Author:** ![Sylvain-69](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain-69/32/26561_2.png) [@Sylvain-69](https://discuss.elastic.co/u/Sylvain-69)\ **Post date:** [January 22, 2018, 9:06pm UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/10 "2018-01-22T21:06:24Z")
Here is the input : [https://gist.github.com/Sylvain-69/6a1e82bcff08e422a6761869f55352e1](https://gist.github.com/Sylvain-69/6a1e82bcff08e422a6761869f55352e1) the filters : - 01 : [https://gist.github.com/Sylvain-69/045a3e13d297c9037239422c578939aa](https://gist.github.com/Sylvain-69/045a3e13d297c9037239422c578939aa) - 02 : [https://gist.github.com/Sylvain-69/42b7e71b97c3d918f7721d59df942b06](https://gist.github.com/Sylvain-69/42b7e71b97c3d918f7721d59df942b06) - 03 : [https://gist.github.com/Sylvain-69/340a48980aa5d227d7e8529ce3929f99](https://gist.github.com/Sylvain-69/340a48980aa5d227d7e8529ce3929f99) - 04 : [https://gist.github.com/Sylvain-69/d8c0d7ac60401105a78090e3307766f7](https://gist.github.com/Sylvain-69/d8c0d7ac60401105a78090e3307766f7) and th output : [https://gist.github.com/Sylvain-69/719f369644f4ed8b4d0f35170cb5ead9](https://gist.github.com/Sylvain-69/719f369644f4ed8b4d0f35170cb5ead9) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 22, 2018, 9:41pm UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/11 "2018-01-22T21:41:02Z")
May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster? | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet... > **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)** > > Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser... ---
**Author:** ![Sylvain-69](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain-69/32/26561_2.png) [@Sylvain-69](https://discuss.elastic.co/u/Sylvain-69)\ **Post date:** [January 23, 2018, 5:07am UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/12 "2018-01-23T05:07:42Z")
Very, very useful links ! Thank you very much ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 20, 2018, 5:08am UTC](https://discuss.elastic.co/t/elk-limits-config-tips-high-cpu-usage-cluster-newbie/116410/13 "2018-02-20T05:08:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.