Packetbeat CPU high


(Guy Wicks) #1

Hi, we've just installed Packetbeat (and topbeat) on a fairly busy (5 to 10 million requests a day) Red Hat webserver (Apache)

Since installing, the CPU / load average has increased quite dramatically, previously the CPU was quite low.

top - 08:38:36 up 647 days, 23:00,  2 users,  load average: 1.18, 0.98, 0.98
Tasks: 496 total,   1 running, 495 sleeping,   0 stopped,   0 zombie
Cpu0  : 48.5%us, 31.9%sy,  0.0%ni, 14.6%id,  0.0%wa,  0.0%hi,  5.0%si,  0.0%st
Cpu1  : 24.4%us,  8.9%sy,  0.0%ni, 65.0%id,  1.3%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   8062644k total,  7786292k used,   276352k free,   311748k buffers
Swap:  6160376k total,    86656k used,  6073720k free,  4060036k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
**30653 root      20   0  561m 116m 7372 S 92.1  1.5 922:15.16 packetbeat**
30511 root      20   0  637m  12m 3024 S 13.2  0.2  40:57.48 topbeat
30447 root      20   0  342m  24m 2160 S  1.6  0.3  41:12.31 filebeat
 9560 monitori  20   0 15304 1548  912 R  1.3  0.0   0:21.69 top
12389 apache    20   0  280m  16m 3892 S  0.7  0.2   0:00.11 httpd
 7451 root      20   0  4120  472  384 S  0.3  0.0   7:44.35 cronolog
 7452 root      20   0  4120  492  384 S  0.3  0.0   7:43.15 cronolog
10502 apache    20   0  280m  19m 3944 S  0.3  0.2   0:00.63 httpd
10982 apache    20   0  280m  20m 3980 S  0.3  0.3   0:00.45 httpd
11295 apache    20   0  280m  19m 3384 S  0.3  0.2   0:00.27 httpd

`

You can see how the CPU has jumped up after installing PB yesterday.

I'm using totally default packetbeat.yml with the only change being to send the output to an Elasticsearch cluster.

Any thoughts / reasons why PB is behaving this way?

G


(Tudor Golubenco) #2

Indeed it seems quite high, compared to everything else on the system being pretty much idle. Can you post your configuration file? Also it would be interesting to see packet statistics on the monitored interface.


(Guy Wicks) #3

As requested, my packetbeat.yml

http://pastebin.com/PGFecsJ0


(Guy Wicks) #4

...having trouble generating reports in Kibaba for packetbeat data - seems to wait a long time and then crash...

What would you like to know about the traffic?


(Guy Wicks) #5


(Guy Wicks) #6

Bytes in/out over the day. Nice peak at 6am!


(Guy Wicks) #7

Number of hit's per second in the last hour:


(Tudor Golubenco) #8

Thanks for the details and sorry for the late answer. The number of requests per second of under 100 shouldn't justify that high CPU. It might be worth removing the protocols that you don't need (just comment them out).

If that doesn't help, you can try to do a bit of profiling.

Start packetbeat using the -httpprof :6060 option, then (optionally on a different machine, becaue you will need go installed), run:

go tool pprof http://localhost:6060/debug/pprof/profile

This will do a profile of 30 seconds. You can than use the interactive command to see what functions are taking most time. I would be interested in the call graph. You can do, for example:

png > /tmp/profile.png

Then post profile.png here.


(Guy Wicks) #9

Thanks for the follow up. I removed everything except HTTP but no significant difference in CPU. I'll run the profile now...


(Tim Arp) #10

Hi,
Did anything get discovered here. I'm using packetbeat to monitor memcache servers. It is maxing out the cpu when I run it, which then impacts memcache performance.

If there is something I should capture let me know. This is 5.4.0 packetbeat


(system) #11