Pipeline-to-pipeline, distributor pattern best practice / known limitation

[yarilc (Yari Latini Corazzini)]

Since I have moved my logging infrastructure to elastic 6.3 I'm evaluating to split and simplifying our existing long chain of 'if' into multiple pipeline-to-pipeline files that end to something like 20 different pipelines.

I have noticed that, by time to time, some of the logstash node fails to start / access one of those pipeline with message like:

"Attempted to send event to 'namedPipeline' but that address was unavailable. Maybe the destination pipeline is down or stopping? Will Retry."

That same configuration is then loaded from other nodes (or after the restart) and, when it occurs, it happens to different pipeline.

Is that a know issue? Am I pushing the pipeline-to-pipeline and the distributor pattern too far with so many pipelines?

I have 1 input pipeline (receving events from filebeat agents), up to 24 processing pipelines and 2 output pipelines (one for elasticsearch and another one for syslog, which is used to deliver some data to a splunk forwarder)

y.

[Kamen_MacKay]

Bump

I'm having the exact same problem (with the distributor pattern). Restarting logstash doesn't seem to 'release' the port. I've setup a beat_server in pipelines.yml using the distributor pattern which collects syslogs from two machines (as a test) and sends them to a syslog pipeline. This would seem to indicate that this isn't load-dependent. A bug has been filed that includes the "address was unavailable" message.

As a side note, I have set my beats server to listen on port 5200 so I ran the below while logstash was running and giving the above error message:

netstat -plaNt | grep 5200
tcp6       0      0 [::]:5200               [::]:*                  LISTEN      25754/java

So it's listening but not accepting connections perhaps?

Restarting filebeat on one of my servers gives me this:

2018-10-12T05:16:42-07:00 INFO Error publishing events (retrying): write tcp 10.2.2.126:48576->10.2.2.226:5200: write: connection reset by peer                                                        │
2018-10-12T05:17:07-07:00 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.call_count