# Winlog beat connection issues

**URL:** <https://discuss.elastic.co/t/winlog-beat-connection-issues/343286>\
**Category:** Beats\
**Tags:** filebeat, winlogbeat\
**Created:** [September 18, 2023, 5:30pm UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286 "2023-09-18T17:30:45Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![JJ007](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jj007/32/124111_2.png) [@JJ007](https://discuss.elastic.co/u/JJ007)\
**Post date:** [September 18, 2023, 5:30pm UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/1 "2023-09-18T17:30:45Z")

</div>

Hi,

We are seeing a large number of connections from winlogbeat to EH Kafka. We have increased the keep alive setting as per the kafka recommendation by MS to 180,000 and also changing the partition setting to random.

Is there anything else that we can do to limit the number of connections.

---

<div class="post-metadata">

**Author:** ![grfneto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grfneto/32/125776_2.png) [@grfneto](https://discuss.elastic.co/u/grfneto)\
**Post date:** [September 18, 2023, 6:04pm UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/2 "2023-09-18T18:04:03Z")

</div>

Hi @JJ007

Have you checked the options in the documentation?

[Configure the Kafka output | Winlogbeat Reference [8.10] | Elastic](https://www.elastic.co/guide/en/beats/winlogbeat/current/kafka-output.html)

---

<div class="post-metadata">

**Author:** ![JJ007](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jj007/32/124111_2.png) [@JJ007](https://discuss.elastic.co/u/JJ007)\
**Post date:** [September 18, 2023, 10:16pm UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/3 "2023-09-18T22:16:07Z")

</div>

Hi @grfneto ,

Yes and this is what I have. The connections doesn't seem to have gone down.

From what I understand, for each transaction (what constitutes the connection), it batches up 2048 events and sends it to Gateway ( EventHub on Kafka protocol).

I see this in the logs:  
},  
"outputs": {  
"kafka": {  
"bytes\_read": 3670,  
"bytes\_write": 662642  
}  
},  
"pipeline": {  
"clients": 32,  
"events": {  
"active": 5,  
"published": 178,  
"total": 178  
},  
"queue": {  
"acked": 174  
}  
}  
}

How do I parse  
"outputs"."kafka".bytes\_read": 3670, - read from the mem queue?  
"outputs"."kafka". "bytes\_write": 662642 - written to consumer?

output.kafka:  
enabled: true  
hosts: ["xyz"]  
topic: "xyz"  
required\_acks: 1  
username: "$ConnectionString"  
password: \<removed?  
compression: none  
ssl.enabled: true  
partition.random:  
reachable\_only: false  
keep\_alive: 180000

#### `max_message_bytes` is set to 1MB as per the capacity of broker.

I can increase `bulk_flush_frequency` but this will introduce latency. In order to make a good estimate need to know the size of the packet/events that is sent. And I am hoping to get that answer from logs?

---

<div class="post-metadata">

**Author:** ![JJ007](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jj007/32/124111_2.png) [@JJ007](https://discuss.elastic.co/u/JJ007)\
**Post date:** [September 19, 2023, 3:33pm UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/4 "2023-09-19T15:33:42Z")

</div>

I have bulk\_max\_size set to 3500. So from the docs, I understand if there are more than the configured # of events, beat agent will batch is out and have this sent in batches.

But from what I see in the logs, there is a total of 235 events and 20 batches what am i missing here?  
Shouldn't the batch count be 0?

libbeat: { [-]  
config: { [+]  
}  
output: { [-]  
events: { [-]  
acked: 250  
active: 0  
batches: 20  
total: 235  
}  
}  
outputs: { [-]  
kafka: { [-]  
bytes\_read: 6070  
bytes\_write: 943619  
}  
}  
pipeline: { [-]  
clients: 32  
events: { [-]  
active: 9  
published: 244  
total: 244  
}  
queue: { [-]  
acked: 250

---

<div class="post-metadata">

**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\
**Post date:** [September 21, 2023, 3:44am UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/5 "2023-09-21T03:44:39Z")

</div>

Even though you have set the `bulk_max_size` to `3500`, it doesn't mean that every batch will contain 3500 events. If, for instance, Filebeat has only collected 500 events, then it will send a batch of 500 events. From the log snippet you shared, the `batches: 20` entry under `output -> events` suggests that 20 batches of events have been sent. However, this doesn't provide details on the size of each batch. You can use the `bulk_flush_frequency` to increase per-flush efficiency.

Generally, in Kafka, you should see N connections (with N being the number of brokers assigned to a topic) plus an additional one for metadata, per producer. This is a simplification, as the actual connection dynamics can be more complex, but it provides a basic idea. I understand you are using eventHub and levering Kafka client.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 19, 2023, 5:44am UTC](https://discuss.elastic.co/t/winlog-beat-connection-issues/343286/6 "2023-10-19T05:44:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.