TCP Input - behaviour under load

[paulkeogh]

I was looking at the method in the TCP input plugin that enqueues incoming documents onto the output queue.

def enqueue_decorated(event, client_ip_address, client_address, client_port, socket)
event.set(HOST_FIELD, client_address) unless event.get(HOST_FIELD)
event.set(HOST_IP_FIELD, client_ip_address) unless event.get(HOST_IP_FIELD)
event.set(PORT_FIELD, client_port) unless event.get(PORT_FIELD)
event.set(SSLSUBJECT_FIELD, socket.peer_cert.subject.to_s) if socket && @ssl_enable && @ssl_verify && event.get(SSLSUBJECT_FIELD).nil?
decorate(event)
@output_queue << event

end

What happens if output_queue is full and the << operator fails ? Does the TCP client get any notification ?

[yaauie (Ry Biesemeyer)]

Both queue implementations that ship with Logstash (memory and persistent) will block the input's thread when an event is pushed onto the queue and the queue does not have capacity, until such a time as the event has been added to the queue. In the TCP input's case, once the thread is blocked, it will stop reading from the socket, applying back-pressure to the sender at the protocol level.