I'm Using TCP plugin inside input and opened port => 12345 and then I've started logstash which works a listener, on the other side I've generated a sample shell script which works as a client and opened a TCP port 12345 and sending a message to the opened port, so far I've achieved 800 RPS with this approach BUT I want to hit max number of messages which can be 50K RPS, below is the script I've used for this
#!/bin/bash
do nc localhost 12345 < /Users/vaseem/Desktop/sh.log in a while block
and below is my .conf file
input{
tcp{
port => 12345
}
}
output{
stdout{codec=>rubydebug}
}
This is the approach I've followed, can anyone please tell me any better approach to hit 50K records per second to the opened tcp port.
You are not measuring how fast logstash can receive messages, you are measuring how fast /bin/bash can fork and exec copies of netcat.
For a command like cat /tmp/file.txt > /dev/null which does not write to disk I can get up to about 1200 calls per second. For echo foo > /tmp/file.txt I find it is around 800 calls per second.
Write a little program to make a socket connection and send the messages. There is a sample perl client here, but you can use any language you are comfortable with.
Yeah other than shell I've used a sample python script as well, but logstash stops listening it after few seconds and it returns an error like below..
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/Users/vaseem/Desktop/m_proc.py", line 9, in publish_messages
clientSocket.connect(("0.0.0.0",12345));
ConnectionResetError: [Errno 54] Connection reset by peer
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "m_proc.py", line 18, in <module>
pool.map(publish_messages, range(1,100))
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/pool.py", line 771, in get
raise self._value
ConnectionResetError: [Errno 54] Connection reset by peer
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.