# Http input does not result in the expected number of documents **URL:** \ **Category:** Logstash\ **Created:** [May 3, 2022, 3:32am UTC](https://discuss.elastic.co/t/http-input-does-not-result-in-the-expected-number-of-documents/303797 "2022-05-03T03:32:49Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![hughjarse](https://avatars.discourse-cdn.com/v4/letter/h/47e85d/32.png) [@hughjarse](https://discuss.elastic.co/u/hughjarse)\ **Post date:** [May 3, 2022, 3:32am UTC](https://discuss.elastic.co/t/http-input-does-not-result-in-the-expected-number-of-documents/303797/1 "2022-05-03T03:32:49Z")
Hi. I would like ideas about what Logstash functionality I can use to troubleshoot a problem I'm experiencing with the http input plugin. ### Problem My http input does not seem to receive?/process? the expected number of events that I am (nearly) certain are being sent to that input. Because there are many components in the log-sending chain, I want to determine if Logstash is the problem. I can't tell if Logstash: - is refusing some network-layer connections to the input - is too busy to accept input data - is dropping events due to filter problems - is unable to send some output - any combination of the above. No errors are reported in the Logstash logs (not using debug). The CPU/memory/network performance of the Logstash server hardware seems completely normal. I'm hoping that Logstash itself can provide some info/metrics which might help me determine whether or not Logstash is the cause of the problem. ### Background I have a third-party web service which sends log events via webhooks. The service is configured to send to 2 destinations simultaneously. For now, I have to assume that the service is sending identical data to the two destinations. Destination1 is a third-party cloud-hosted Elastic stack ServiceX (the webhook data is sent directly). Destination2 is my own AWS-hosted Logstash instance with an http input. The data is enriched by Logstash and then forwarded on to ServiceX. This Logstash instance is also receiving/filtering multiple other kinds of input from other external sources (mostly Filebeat). Regardless of the time period or even data volume, webhook data sent via Destination2 (my self-hosted Logstash) consistently results in only between 1/3 to 1/2 the number of documents compared to the identical webhook data sent directly to ServiceX. I assume that ServiceX utilizes a queue service when they receive data, whereas my own Logstash does not. This is a problem because I **much** prefer to put **only** my enriched Destination2 data into Elasticsearch. Previously, Destination2 had an AWS Network Load Balancer in front of it which accepted TLS connections and then forwarded them to Logstash's non-TLS http input. In an attempt to troubleshoot this issue, I re-configured the NLB to be a simple TCP listener which forwarded connections to Logstash's TLS-configured http input. This actually made the problem worse, with "lost" documents increasing from roughly 1/2 to 2/3. This is what leads me to believe that Logstash is the bottleneck in my chain. Any help would be most welcome. 😀 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 31, 2022, 3:32am UTC](https://discuss.elastic.co/t/http-input-does-not-result-in-the-expected-number-of-documents/303797/2 "2022-05-31T03:32:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.