TCP plugin tuning

[vaseemQA (Vaseem Akram Mohammad)]

Hi Team,

I'm Using TCP plugin inside input and opened port => 12345 and then I've started logstash which works a listener, on the other side I've generated a sample shell script which works as a client and opened a TCP port 12345 and sending a message to the opened port, so far I've achieved 800 RPS with this approach BUT I want to hit max number of messages which can be 50K RPS, below is the script I've used for this

#!/bin/bash
do nc  localhost 12345 <  /Users/vaseem/Desktop/sh.log in a while block

and below is my .conf file

input{
    tcp{
        port => 12345
    }
}
output{
stdout{codec=>rubydebug}
}

This is the approach I've followed, can anyone please tell me any better approach to hit 50K records per second to the opened tcp port.

Thanks in Advance,
Vaseem.

[leandrojmp (Leandro Pereira)]

I don't think there is any kind of tuning on this plugin, it just listens to the port.

The number of events will depend on other factors related to the sender and the operating system.

Also, where did those e/s numbers came from? Why 50k e/s? It is a requirement? How are you measuring the events per second rate?

[Badger]

You are not measuring how fast logstash can receive messages, you are measuring how fast /bin/bash can fork and exec copies of netcat.

For a command like cat /tmp/file.txt > /dev/null which does not write to disk I can get up to about 1200 calls per second. For echo foo > /tmp/file.txt I find it is around 800 calls per second.

Write a little program to make a socket connection and send the messages. There is a sample perl client here, but you can use any language you are comfortable with.

[vaseemQA (Vaseem Akram Mohammad)]

@Badger

receive

Hey @leandrojmp Thank you for the response, I want to analyse how does logstash works at 50k e/s
And I'm measuring it given endpoint

localhost:9600/_node/stats/pipelines?pretty

and the response of it is

{
  "host" : "AMAC02FF2ETMD6P",
  "version" : "6.8.3",
  "http_address" : "127.0.0.1:9600",
  "id" : "cb93ce6b-f487-41a2-bd3d-834456d7dfa4",
  "name" : "AMAC02FF2ETMD6P",
  "pipelines" : {
    "main" : {
      "events" : {
        "in" : 16287,
        "out" : 16287,
        "duration_in_millis" : 97903,
        "queue_push_duration_in_millis" : 22670,
        "filtered" : 16287
      },
      "plugins" : {
        "inputs" : [ {
          "id" : "ca4618847b10c4e385f86017955bcf165988002e792bc4fed2b1e80a100162f2",
          "events" : {
            "out" : 16287,
            "queue_push_duration_in_millis" : 22670
          },
          "name" : "tcp"
        } ],
        "filters" : [ ],
        "outputs" : [ {
          "id" : "4c68340d1e19c496d5b3a990e2fcb9a752a44b6b32f4441acfef4aa3a31aa99c",
          "events" : {
            "in" : 16287,
            "out" : 16287,
            "duration_in_millis" : 97639
          },
          "name" : "stdout"
        } ]
      },
      "reloads" : {
        "successes" : 0,
        "last_success_timestamp" : null,
        "failures" : 0,
        "last_failure_timestamp" : null,
        "last_error" : null
      },
      "queue" : {
        "type" : "memory"
      }
    }
  }
}

And this is the response I got when I've used a python script.

Thanks,
Vaseem

[vaseemQA (Vaseem Akram Mohammad)]

Hi @Badger ,

Yeah other than shell I've used a sample python script as well, but logstash stops listening it after few seconds and it returns an error like below..

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/Users/vaseem/Desktop/m_proc.py", line 9, in publish_messages
    clientSocket.connect(("0.0.0.0",12345));
ConnectionResetError: [Errno 54] Connection reset by peer
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "m_proc.py", line 18, in <module>
    pool.map(publish_messages, range(1,100))
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 364, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
ConnectionResetError: [Errno 54] Connection reset by peer

Thanks in Advance

Vaseem

[Badger]

What does logstash log when the python script gets that error?