ELK Limits, config tips, high CPU usage, cluster (newbie)

Elastic Stack Elasticsearch
1

Hello Everybody !

I started to use ELK in my company, at first to be able to get a look at windows logon informations then members of security groups activity, etc...

I installed one node on Centos 7 (VM) with 2 CPUs (Virtual sockets) and 8 Go RAM.

ELK was installed with the default configuration, I only set the basics to get it work.

Of course after digging a little bit, I found out ELK to be very usefull, so I started to send more logs into it (netflow, winlogbeat), and now we are asked to monitor network activity on some PCs (so I used Packetbeat), etc...

I did manage for winlogbeat and packbeat to send to ELK only the necessary (only the events needed and only the traffic from/to the IPs needed) , but recently I added 2 more netflow logs and now my node is at 100% CPU usage, it seems that it is to much for it...

Since it's in "production" and that it will grow because I want to developp the use of ELK in my company.

I need advices on my configuration and what to do in the futur (cluster, how many nodes ? do I need Redis ? etc...).

I need to know the best practice for the use of netflow codec, csv, geoip, etc and other filters, If there is some tools to monitor logstash I/O, and if you have some tips and how you guys work with ELK to get it working fast and efficient (or any other tips that is usefull), if there is some kind of limits to logstash or elasticsearch ?

I know that there is a lot of informations on internet and that google is my best friend, but I already spent a lot of time starting from zero to get where I am, and sometimes it's hard to understand everything I read... So maybe if we can discuss about that it will help me understand better and found the informations I need to evolve !

At the same time if you have some links that can be good for a newbie to read, it will be appreciated !

thanks !

8

Bonjour :wink:

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

For big part of code, you can use gist.github.com to share it.

Would be nice if you edit your post (move main stuff to gist and delete the part of the post which are not needed anymore).

Also FYI there is a french section at #in-your-native-tongue:discussions-en-francais but feel free to continue posting here in english. Just please use English (some comments are in french). :slight_smile:

1 Like
10

Here is the input : https://gist.github.com/Sylvain-69/6a1e82bcff08e422a6761869f55352e1

the filters :

and th output : https://gist.github.com/Sylvain-69/719f369644f4ed8b4d0f35170cb5ead9

11

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

1 Like
12

Very, very useful links !

Thank you very much !

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.