I was doing perf test to Logstash's network inputs, UDP/TCP. I wrote a python script to simulate some messages generator from network. The python script and Logstash is located in different machines in LAN. So the setup looks like this:
python socket->tcp/udp->logstash->local file
After some rough tests, I observed that at least in my method of testing, UDP drops lots of messages and TCP works fine with an acceptable performance (10 clients each send 100k msg in 37s).
My testing python script is attached below. I only tried nc -4u but also observed messages dropped. Still investigating if this is caused by Logstash input or the way I send message through UDP.
def udp_worker(port):
start = time.time()
sock = socket.socket(socket.AF_INET,
socket.SOCK_DGRAM)
print('Begin sendding data to port %d' % port)
retval = 0
for i in range(0, MSG_AMOUNT):
retval += sock.sendto(MESSAGE_BASE % (i, port, randrange(100)), (UDP_IP, port))
print('Total amount of data sent %d in time %s' % (retval, str(time.time() - start)))
def tcp_worker(port):
start = time.time()
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
sock.connect((UDP_IP, UDP_BASE_PORT))
print('Begin sendding data to port %d' % port)
for i in range(0, MSG_AMOUNT):
sock.sendall(MESSAGE_BASE % (i, port, randrange(100)) )
#data = sock.recv(1024)
sock.close()
print('Total in time %s' % str(time.time() - start))
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
MSG_AMOUNT = int(sys.argv[1])
if len(sys.argv) > 2:
PROCESS = int(sys.argv[2])
workers = deque()
for i in range(0, PROCESS):
port = UDP_BASE_PORT
#t = threading.Thread(target = udp_worker, args = [port]) # comments this for testing tcp only
t = threading.Thread(target = tcp_worker, args = [port])
t.start()
print("%s start" % t)
workers.append(t)
for w in workers:
print("%s wait for join" % w)
w.join()
I appreciate if some one can provide a stable environment transferring data through UDP or point out anything I might miss during my test!
Btw, I didn't see things related to UDP input in 1.5.0 change log.
EDIT: forgot to post my Logstash settings
input {
udp/tcp {
port => xxxx
}
}
output {
file {
path => "/tmp/tmpfile"
}
}