Lots of Documents getting failed and dropped at the Logstash(seen in logstash logs)

Elastic Search
1

We use logstash to send the logs to Azure EventHub and then to Azure Data Explorer(ADX). Also, we use Filebeat to send the logs to logstash first. The issue is that some of the logs are not reaching to ADX and its intermittent. During troubleshooting we found that, Filebeat is parsing all those logs and sending to logstash but then logs are somehow getting dropped at the Logstash level which we are observing quite frequently in the logstash logs as shown below. I dont know if this is the actual issue or not but logs are indeed getting lost in the middle.
Do you feel we can make some config changes at the logstash.yaml or logstash.conf file to avoid this issue ?

[2026-01-13T09:39:44,275][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>332190196, :documents_successful=>332185835, :documents_failed=>383, :documents_dropped=>0, :batches_successful=>1527425, :batches_failed=>1}
[2026-01-13T09:39:44,286][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>1978372787, :documents_successful=>1978370297, :documents_failed=>0, :documents_dropped=>3254, :batches_successful=>4996783, :batches_failed=>0}

2

Hello and welcome,

Do you have other error logs?

3

Hello @leandrojmp - These error logs were taken from logstash-plain.log. I dont see any other errors apart from these kind of errors.

[2026-01-14T12:56:34,029][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1320504, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:56:44,767][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1174632, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:56:45,151][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>3312641667, :documents_successful=>3312632963, :documents_failed=>0, :documents_dropped=>4748, :batches_successful=>8355091, :batches_failed=>0}
[2026-01-14T12:56:45,151][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>548142809, :documents_successful=>548137204, :documents_failed=>383, :documents_dropped=>0, :batches_successful=>2507758, :batches_failed=>1}
[2026-01-14T12:57:05,879][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1169284, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:57:05,889][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1461570, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:57:06,325][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1169284, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:57:45,151][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>3313537880, :documents_successful=>3313530475, :documents_failed=>0, :documents_dropped=>4751, :batches_successful=>8357336, :batches_failed=>0}
[2026-01-14T12:57:45,151][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>548275222, :documents_successful=>548270334, :documents_failed=>383, :documents_dropped=>0, :batches_successful=>2508366, :batches_failed=>1}
[2026-01-14T12:58:45,152][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>3314395538, :documents_successful=>3314389170, :documents_failed=>0, :documents_dropped=>4751, :batches_successful=>8359557, :batches_failed=>0}
[2026-01-14T12:58:45,152][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>548408093, :documents_successful=>548405028, :documents_failed=>383, :documents_dropped=>0, :batches_successful=>2508990, :batches_failed=>1}
[2026-01-14T12:59:18,092][ERROR][logstash.outputs.eventhubs][main][logs_to_eventhubs] Document exceeds max size {:size=>1465830, :max_allowed=>1037576, :truncated=>true}
[2026-01-14T12:59:45,152][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>548540221, :documents_successful=>548536297, :documents_failed=>383, :documents_dropped=>0, :batches_successful=>2509602, :batches_failed=>1}
[2026-01-14T12:59:45,152][INFO ][logstash.outputs.eventhubs][main] EventHubs Metrics {:documents_queued=>3315351555, :documents_successful=>3315347816, :documents_failed=>0, :documents_dropped=>4752, :batches_successful=>8361902, :batches_failed=>0

Please suggest us what could be causing it and what config changes might fix it ?
I am using all default values in logstash.yml with just the below active settings :

pipeline.workers: 30