# Assistance Needed with TCP Socket Log Transmission to Elastic

**URL:** <https://discuss.elastic.co/t/assistance-needed-with-tcp-socket-log-transmission-to-elastic/358192>\
**Category:** Elasticsearch\
**Created:** [April 25, 2024, 9:28am UTC](https://discuss.elastic.co/t/assistance-needed-with-tcp-socket-log-transmission-to-elastic/358192 "2024-04-25T09:28:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kamlesh.Solanki](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kamlesh.solanki/32/133839_2.png) [@Kamlesh.Solanki](https://discuss.elastic.co/u/Kamlesh.Solanki)\
**Post date:** [April 25, 2024, 9:28am UTC](https://discuss.elastic.co/t/assistance-needed-with-tcp-socket-log-transmission-to-elastic/358192/1 "2024-04-25T09:28:54Z")

</div>

Subject: Assistance Needed with TCP Socket Log Transmission to Elastic

Hello Elastic Community,

I'm encountering an issue with sending logs to Elastic's Netskope integration using the Python `socket` library, and I'd appreciate some guidance.

### Workflow Overview

Here's a brief outline of my current process:

1. **Log Collection:** I collect logs from a Netskope Tenant.
2. **Data Processing:** The logs are parsed into ECS (Elastic Common Schema) format, compatible with the Netskope integration.
3. **Socket Connection:** I establish a TCP socket connection to the Elastic server.
4. **Data Splitting:** I separate multiple events with newline characters (`\n`) and create a JSON dump for the payload.
5. **Data Transmission:** I send alerts/events to the Elastic server using the `sendall` method from the `socket` library.

### TCP Connection Code

This is how I establish the TCP connection:

```python
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.connect(
    (<Server IP>, <Server Port>)
)

```

### Sending Logs Code

```auto
self.sock.sendall(bytes(data, encoding="utf-8"))

```

### The Issue

I'm sending batches of up to 10,000 logs at a time, and it generally works. However, when I increase the rate to 100,000 EPM (Events Per Minute), some logs get truncated or lost. Here are the error messages I'm seeing in Elastic:

 ![Error in Elastic](https://us1.discourse-cdn.com/elastic/original/3X/7/c/7c0ef654e7a7167e9f8ba1aa9bb8d955564af5d0.png)

### Questions

1. **Elastic's Ingestion Rate** : Is there a limit on how fast Elastic can accept data? If so, what's the recommended rate to avoid data loss or truncation?
2. **TCP Socket Best Practices** : Is my approach of sending logs in large batches through a single socket connection valid, or could it be causing the issue? Should I consider creating a new socket for each event?
3. **Optimal Approach** : What are the best practices for sending high-throughput logs to Elastic via TCP socket? Would using a different tool like Logstash or Beats be a better option?

I'd really appreciate any advice or best practices on addressing this issue and ensuring reliable log transmission to Elastic.

Thank you in advance for your help!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2024, 9:29am UTC](https://discuss.elastic.co/t/assistance-needed-with-tcp-socket-log-transmission-to-elastic/358192/2 "2024-05-23T09:29:10Z")

</div>

Closed on May 23, 2024, 9:29am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.