# Debug Unmatched responses **URL:** \ **Category:** Beats\ **Tags:** packetbeat\ **Created:** [February 14, 2018, 6:27pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862 "2018-02-14T18:27:03Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![lren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lren/32/27269_2.png) [@lren](https://discuss.elastic.co/u/lren)\ **Post date:** [February 14, 2018, 6:27pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/1 "2018-02-14T18:27:03Z")
Hi, I'm noticing a lot of http requests that aren't being sent through to logstash via packetbeat. When I stop packetbeat I notice a ton of http.unmatched\_responses and tcp.dropped\_because\_of\_gaps: ``` INFO Total non-zero values: beat.info.uptime.ms=687480148 beat.memstats.gc_next=171292832 beat.memstats.memory_alloc=115671728 beat.memstats.memory_total=62745787971712 http.unmatched_responses=17974428 libbeat.config.module.running=0 libbeat.output.events.acked=632178 libbeat.output.events.batches=224225 libbeat.output.events.failed=341 libbeat.output.events.total=632519 libbeat.output.read.bytes=1343814 libbeat.output.type=logstash libbeat.output.write.bytes=297294233 libbeat.output.write.errors=256 libbeat.pipeline.clients=13 libbeat.pipeline.events.active=0 libbeat.pipeline.events.filtered=93530152 libbeat.pipeline.events.published=632178 libbeat.pipeline.events.retry=722 libbeat.pipeline.events.total=94162330 libbeat.pipeline.queue.acked=632178 tcp.dropped_because_of_gaps=9308590 2018/02/14 16:51:51.724844 metrics.go:52: INFO Uptime: 190h58m0.486932728s ``` Configs are pretty close to the default. Any ideas on what could be causing this or how to approach debugging this issue? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 14, 2018, 6:36pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/2 "2018-02-14T18:36:04Z")
You'll probably need to due some tuning if you are on the defaults. In particular switch away from using pcap to af\_packet. See [https://www.elastic.co/guide/en/beats/packetbeat/6.2/configuration-interfaces.html](https://www.elastic.co/guide/en/beats/packetbeat/6.2/configuration-interfaces.html) There was some good discussion in this ticket and the linked and the ticket it links to: [Performance of Packetbeat?](https://discuss.elastic.co/t/performance-of-packetbeat/116022) ---
**Author:** ![lren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lren/32/27269_2.png) [@lren](https://discuss.elastic.co/u/lren)\ **Post date:** [February 15, 2018, 5:02pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/3 "2018-02-15T17:02:50Z")
Thanks. I read through that, but don't think my situation has anything to do with an outdated kernel. Our requests are generally larger than what you would probably typically expect (anywhere from 0-30mb) and responses could take multiple seconds. Would increasing buffer\_size\_mb directly help our situation? Or potentially tweaking our internal queue memory to flush more frequently? Is there anything else that could be tuned to not lose packets? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 15, 2018, 6:39pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/4 "2018-02-15T18:39:34Z")
Did you switch to af\_packet? I would try that with an increased buffer\_size\_mb value. I would expect this to help with `tcp.dropped_because_of_gaps`. However if the individual request or response is greater than 10MB Packetbeat will drop the transaction. You can increase the limit by setting [max\_message\_size](https://www.elastic.co/guide/en/beats/packetbeat/6.2/packetbeat-http-options.html#_literal_max_message_size_literal). If you enable debug for `http` you should see a log message anytime this happens. > ---
**Author:** ![lren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lren/32/27269_2.png) [@lren](https://discuss.elastic.co/u/lren)\ **Post date:** [February 15, 2018, 8:24pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/5 "2018-02-15T20:24:14Z")
Yep, I switched to af\_packet and set the buffer\_size\_mb to 4028. I think I've discovered the issue to be the include\_body\_for... I currently have this set under http protocol: ``` packetbeat.protocols: - type: http # Configure the ports where to listen for HTTP traffic. You can disable # the HTTP protocol by commenting out the list of ports. ports: [9200] send_request: true send_response: false include_body_for: ["application/json", "x-www-form-urlencoded"] transaction_timeout: 5s ``` When I remove the include\_body\_for, I can see all traffic coming through with no issue. But as soon as I enable it, my performance tanks. I believe it's because our response bodies are huge and causes Packetbeat to be unable to keep up? Is there a way to only include the request body (I just want to see the incoming query for Elasticsearch where Packetbeat is installed)? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 16, 2018, 8:51pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/6 "2018-02-16T20:51:16Z")
> [@lren](#): > > Is there a way to only include the request body (I just want to see the incoming query for Elasticsearch where Packetbeat is installed)? AFAICT there isn't way to choose to only include one or the other. That seems like a reasonable feature to have if you want to open an enhancement request. I was also wondering if you have disabled flows since you only want to log ES queries? Have you set `packetbeat.flows.enabled: false`? That should allow Packetbeat to set a more narrow BPF filter. ---
**Author:** ![lren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lren/32/27269_2.png) [@lren](https://discuss.elastic.co/u/lren)\ **Post date:** [February 16, 2018, 9:09pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/7 "2018-02-16T21:09:08Z")
I currently have flows enabled. This is mostly to handle the case where a really long running query comes in. I don't want it to be stuck in the buffer for super long, so I have a fairly small timeout (3sec right now). Instead, I just want to log it and cap it at X seconds (behaving somewhat similar to a slow query log). Is this the best way to do that? Or is there somewhere else to do this that would be more efficient? ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [February 16, 2018, 11:21pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/8 "2018-02-16T23:21:49Z")
If you are only interested in traffic on 9200 then I suggest using a custom BPF filter while still having flows enabled. `packetbeat.interfaces.bpf_filter: tcp port 9200` This will stop any other traffic from being processed by Packetbeat. This might help some. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 16, 2018, 11:22pm UTC](https://discuss.elastic.co/t/debug-unmatched-responses/119862/9 "2018-03-16T23:22:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.