Packetbeat CPU high

Elastic Stack Beats
1

Hi, we've just installed Packetbeat (and topbeat) on a fairly busy (5 to 10 million requests a day) Red Hat webserver (Apache)

Since installing, the CPU / load average has increased quite dramatically, previously the CPU was quite low.

top - 08:38:36 up 647 days, 23:00,  2 users,  load average: 1.18, 0.98, 0.98
Tasks: 496 total,   1 running, 495 sleeping,   0 stopped,   0 zombie
Cpu0  : 48.5%us, 31.9%sy,  0.0%ni, 14.6%id,  0.0%wa,  0.0%hi,  5.0%si,  0.0%st
Cpu1  : 24.4%us,  8.9%sy,  0.0%ni, 65.0%id,  1.3%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   8062644k total,  7786292k used,   276352k free,   311748k buffers
Swap:  6160376k total,    86656k used,  6073720k free,  4060036k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
**30653 root      20   0  561m 116m 7372 S 92.1  1.5 922:15.16 packetbeat**
30511 root      20   0  637m  12m 3024 S 13.2  0.2  40:57.48 topbeat
30447 root      20   0  342m  24m 2160 S  1.6  0.3  41:12.31 filebeat
 9560 monitori  20   0 15304 1548  912 R  1.3  0.0   0:21.69 top
12389 apache    20   0  280m  16m 3892 S  0.7  0.2   0:00.11 httpd
 7451 root      20   0  4120  472  384 S  0.3  0.0   7:44.35 cronolog
 7452 root      20   0  4120  492  384 S  0.3  0.0   7:43.15 cronolog
10502 apache    20   0  280m  19m 3944 S  0.3  0.2   0:00.63 httpd
10982 apache    20   0  280m  20m 3980 S  0.3  0.3   0:00.45 httpd
11295 apache    20   0  280m  19m 3384 S  0.3  0.2   0:00.27 httpd

`

Pasted image951×703 57.7 KB

You can see how the CPU has jumped up after installing PB yesterday.

I'm using totally default packetbeat.yml with the only change being to send the output to an Elasticsearch cluster.

Any thoughts / reasons why PB is behaving this way?

G

2

Indeed it seems quite high, compared to everything else on the system being pretty much idle. Can you post your configuration file? Also it would be interesting to see packet statistics on the monitored interface.

3

As requested, my packetbeat.yml

http://pastebin.com/PGFecsJ0

4

...having trouble generating reports in Kibaba for packetbeat data - seems to wait a long time and then crash...

What would you like to know about the traffic?

5

Pasted image1200×891 114 KB

6

Bytes in/out over the day. Nice peak at 6am!

Pasted image1153×899 131 KB

7

Number of hit's per second in the last hour:

Pasted image1194×851 118 KB

8

Thanks for the details and sorry for the late answer. The number of requests per second of under 100 shouldn't justify that high CPU. It might be worth removing the protocols that you don't need (just comment them out).

If that doesn't help, you can try to do a bit of profiling.

Start packetbeat using the -httpprof :6060 option, then (optionally on a different machine, becaue you will need go installed), run:

go tool pprof http://localhost:6060/debug/pprof/profile

This will do a profile of 30 seconds. You can than use the interactive command to see what functions are taking most time. I would be interested in the call graph. You can do, for example:

png > /tmp/profile.png

Then post profile.png here.

9

Thanks for the follow up. I removed everything except HTTP but no significant difference in CPU. I'll run the profile now...

10

Hi,
Did anything get discovered here. I'm using packetbeat to monitor memcache servers. It is maxing out the cpu when I run it, which then impacts memcache performance.

If there is something I should capture let me know. This is 5.4.0 packetbeat