ELK Limits, config tips, high CPU usage, cluster (newbie)


(Sylvain 69) #1

Hello Everybody !

I started to use ELK in my company, at first to be able to get a look at windows logon informations then members of security groups activity, etc...

I installed one node on Centos 7 (VM) with 2 CPUs (Virtual sockets) and 8 Go RAM.

ELK was installed with the default configuration, I only set the basics to get it work.

Of course after digging a little bit, I found out ELK to be very usefull, so I started to send more logs into it (netflow, winlogbeat), and now we are asked to monitor network activity on some PCs (so I used Packetbeat), etc...

I did manage for winlogbeat and packbeat to send to ELK only the necessary (only the events needed and only the traffic from/to the IPs needed) , but recently I added 2 more netflow logs and now my node is at 100% CPU usage, it seems that it is to much for it...

Since it's in "production" and that it will grow because I want to developp the use of ELK in my company.

I need advices on my configuration and what to do in the futur (cluster, how many nodes ? do I need Redis ? etc...).

I need to know the best practice for the use of netflow codec, csv, geoip, etc and other filters, If there is some tools to monitor logstash I/O, and if you have some tips and how you guys work with ELK to get it working fast and efficient (or any other tips that is usefull), if there is some kind of limits to logstash or elasticsearch ?

I know that there is a lot of informations on internet and that google is my best friend, but I already spent a lot of time starting from zero to get where I am, and sometimes it's hard to understand everything I read... So maybe if we can discuss about that it will help me understand better and found the informations I need to evolve !

At the same time if you have some links that can be good for a newbie to read, it will be appreciated !

thanks !


(David Pilato) #8

Bonjour :wink:

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

For big part of code, you can use gist.github.com to share it.

Would be nice if you edit your post (move main stuff to gist and delete the part of the post which are not needed anymore).

Also FYI there is a french section at #in-your-native-tongue:discussions-en-francais but feel free to continue posting here in english. Just please use English (some comments are in french). :slight_smile:


(Sylvain 69) #10

Here is the input : https://gist.github.com/Sylvain-69/6a1e82bcff08e422a6761869f55352e1

the filters :

and th output : https://gist.github.com/Sylvain-69/719f369644f4ed8b4d0f35170cb5ead9


(David Pilato) #11

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing


CPU a 100 %, conseils et astuces
(Sylvain 69) #12

Very, very useful links !

Thank you very much !


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.