Hi there!
We are using Elasticsearch and Filebeat to collect logs from our services and servers. Recently we upgraded Elasticsearch from v8.11.3 to v8.13.2.
But after we upgraded Filebeat to the same version we noticed a significant increase in the number of new connections initiated from Filebeats to Elasticsearch cluster. Looks like connections from Filebeats to Elasticsearch cluster are closed much faster than before.
New connections registered on AWS NLB from Filebeats to Elasticsearch cluster
Active connections on AWS NLB
Moreover, the number of new connections increases when Filebeats sends less data which is strange:
I did not notice that this increase affected anything except the libbeat.pipeline.events.failed metric which reports failed events more often, so we see more gaps in logs when the metric reports a value greater than zero.
Not sure that these two things are related, but I would like to clarify if this increase of new connections from Filebeat agents is expected.
Thanks!