# Packetbeat use mem too many when monitor cassandra **URL:** \ **Category:** Beats\ **Tags:** packetbeat\ **Created:** [January 24, 2017, 2:39am UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554 "2017-01-24T02:39:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ymyjohnny](https://avatars.discourse-cdn.com/v4/letter/y/e56c9b/32.png) [@ymyjohnny](https://discuss.elastic.co/u/ymyjohnny)\ **Post date:** [January 24, 2017, 2:39am UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/1 "2017-01-24T02:39:02Z")
when i use packetbeat to monitor cassandra,the packetbeat use too many mem and after a while,the packetbeat process will be dead. version is 5.1.2 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 32614 root 20 0 27.0g 25g 10m S 16.0 20.4 3:02.82 /usr/share/packetbeat/bin/packetbeat -c /etc/packetbeat/packetbeat.yml -path.home /usr/share/packetbeat -path.config /etc/packetbeat -path.data my config packetbeat.interfaces.device: any packetbeat.interfaces.buffer\_size\_mb: 300 packetbeat.flows: timeout: 30s period: 10s packetbeat.protocols.icmp: enabled: true packetbeat.protocols.cassandra: ports: [9042] output.elasticsearch: hosts: ["192.168.32.206"] ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 24, 2017, 2:10pm UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/2 "2017-01-24T14:10:33Z")
do you have some packetbeat logs, and/or pcap file for testing? No idea wether this is a potential leak or incomplete transactions hanging in memory due to packet loss (have you tried af\_packet sniffer?). ---
**Author:** ![ymyjohnny](https://avatars.discourse-cdn.com/v4/letter/y/e56c9b/32.png) [@ymyjohnny](https://discuss.elastic.co/u/ymyjohnny)\ **Post date:** [January 25, 2017, 6:14am UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/3 "2017-01-25T06:14:48Z")
my log 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[stream:19776 op:RESULT length:4 version:3 flags:Default] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:7296 op:RESULT length:248] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[op:RESULT length:4 version:3 flags:Default stream:4992] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:23360 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:32064 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[stream:23040 op:RESULT length:4 version:3 flags:Default] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:19840 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:22208 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[flags:Default stream:29376 op:RESULT length:4 version:3] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:22144 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[length:4 version:3 flags:Default stream:30464 op:RESULT] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[op:RESULT length:4 version:3 flags:Default stream:4480] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[version:3 flags:Default stream:8576 op:RESULT length:4] 2017-01-24T11:31:05+08:00 WARN Response from unknown transaction. Ignoring. map[length:4 version:3 flags:Default stream:25792 op:RESULT] 2017-01-24T11:31:13+08:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.messages\_in\_worker\_queues=8251 libbeat.publisher.published\_events=17254 libbeat.es.published\_and\_acked\_events=17234 libbeat.es.call\_count.PublishEvents=352 libbeat.es.publish.read\_bytes=176526 libbeat.es.publish.write\_bytes=10844063 tcp.dropped\_because\_of\_gaps=2029 2017-01-24T11:31:41+08:00 INFO Non-zero metrics in the last 30s: libbeat.es.published\_and\_acked\_events=27 libbeat.es.publish.read\_bytes=429 libbeat.es.call\_count.PublishEvents=1 libbeat.es.publish.write\_bytes=18371 libbeat.publisher.messages\_in\_worker\_queues=4 libbeat.publisher.published\_events=2051 2017-01-24T11:31:41+08:00 ERR Failed to perform any bulk index operations: Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60562-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:31:41+08:00 INFO Error publishing events (retrying): Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60562-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:31:42+08:00 INFO Connected to Elasticsearch version 5.1.2 2017-01-24T11:31:42+08:00 INFO Trying to load template for client: [http://192.168.32.206:9200](http://192.168.32.206:9200) 2017-01-24T11:31:42+08:00 INFO Template already exists and will not be overwritten. 2017-01-24T11:31:56+08:00 ERR Failed to perform any bulk index operations: Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60719-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:31:56+08:00 INFO Error publishing events (retrying): Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60719-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:31:58+08:00 INFO Connected to Elasticsearch version 5.1.2 2017-01-24T11:31:58+08:00 INFO Trying to load template for client: [http://192.168.32.206:9200](http://192.168.32.206:9200) 2017-01-24T11:31:58+08:00 INFO Template already exists and will not be overwritten. 2017-01-24T11:32:10+08:00 ERR Failed to perform any bulk index operations: Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60720-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:32:10+08:00 INFO Error publishing events (retrying): Post [http://192.168.32.206:9200/\_bulk:](http://192.168.32.206:9200/_bulk:) write tcp 192.168.32.177:60720-\>192.168.32.206:9200: write: connection reset by peer 2017-01-24T11:32:11+08:00 INFO Non-zero metrics in the last 30s: libbeat.es.publish.write\_errors=3 libbeat.es.publish.write\_bytes=448308 libbeat.es.call\_count.PublishEvents=2 libbeat.es.publish.read\_bytes=1096 ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 25, 2017, 5:18pm UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/4 "2017-01-25T17:18:37Z")
There are a number of TCP connections being dropped and incomplete due to packet loss: ```auto tcp.dropped_because_of_gaps=2029 ``` Please to the `af_packet` sniffer, as it's somewhat more performant then the default sniffer. Plus, it seems your having some problems with the output connection getting closed by ES (or proxy). If output get's stuck a number of transactions is kept in queues. ---
**Author:** ![ymyjohnny](https://avatars.discourse-cdn.com/v4/letter/y/e56c9b/32.png) [@ymyjohnny](https://discuss.elastic.co/u/ymyjohnny)\ **Post date:** [January 26, 2017, 2:25am UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/5 "2017-01-26T02:25:48Z")
i has changed config,but mem used 35G PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 77389 root 20 0 39.3g 35g 30m S 99.9 14.4 2:27.86 /usr/share/packetbeat/bin/packetbeat -c /etc/packetbeat/packetbeat.yml -path.home /usr/share/packetbeat -path.config /etc/packetbeat -path.dat packetbeat.interfaces.device: any packetbeat.interfaces.snaplen: 1514 packetbeat.interfaces.type: af\_packet packetbeat.interfaces.buffer\_size\_mb: 100 ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [January 26, 2017, 1:11pm UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/6 "2017-01-26T13:11:55Z")
have you tried to monitor memory usage over time? How long does it take until RSS becomes big (ignore virtual, as runtime just reserves a hughe virtual address space on startup, which actually is no real allocation)? Have you checked your logs, if your still having problems with elasticsearch? you can enable a profiling output in packetbeat by starting packetbeat with `-httpprof localhost:6060` (using `localhost` ensure the endpoint is not reachable from the outside, use `-httpprof :6060` if you want to enable remote debugging). Having the profiling endpoint, you can collect sample of memory allocations via `curl http://localhost:6060/debug/pprof/heap?debug=1`. This call does not add much overhead to packetbeat. If you collect a few traces over time, it let's us see how memory usage 'develops' over time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 23, 2017, 1:12pm UTC](https://discuss.elastic.co/t/packetbeat-use-mem-too-many-when-monitor-cassandra/72554/7 "2017-02-23T13:12:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.