[publisher_pipeline_output] pipeline /output.go:154 Failed to connect to backoff(async(tcp://logstash-xxx-xxx.apps.-xxx.xxx:443)): EOF

[Rohan-boogeyman (Rohan)]

Hello Guys,
Thanks in advance for taking your time to look into the issue i am experiencing.

Error in File beat at the time of Debug
[publisher_pipeline_output] pipeline /output.go:154 Failed to connect to backoff(async(tcp://logstash-xxx-xxx.apps.-xxx.xxx:443)): EOF

When i try to connect to the Logstash instance via Telnet it works fine (I get blank screen) However, at the time of connecting it with Logstash i am getting this error.

Hope to get some assistance on this.

Thanks in Advance!

[Badger]

You are saying that telnet connects to port 443? Is TLS enabled on that port?

[Rohan-boogeyman (Rohan)]

Hi Badger,
For some reason i believed that Port 443 by default has TLS enabled - However if we speak of defaultpipeline.yml - I do have SSL Enabled within that and so it is enabled in Filebeat.yml

Does that help?

[stephenb (Stephen Brown)]

Hi @Rohan-boogeyman

Perhaps you could post your filebeat.yml and the Logstash pipeline for the filebeat input, that would help us understand your configuration.

[Rohan-boogeyman (Rohan)]

Thanks @stephenb !

Please details below - let me know if there's something else you require to assist on this issue.

#================Filebeat Yml Logstash output =============================
output.logstash:
hosts: ["openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com:443"]
ssl.certificate_authorities: ["C:\certs\openshift logstash\xxx_xxx_G2_Intermediate_CA.cer", "C:\certs\openshift logstash\xxx_xxx_G2_Root_CA.cer"]
ssl.certificate: "C:\certs\openshift logstash\openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com.cer"
ssl.key: "C:\certs\openshift logstash\openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx-pkcs8.key"
ssl.enabled: true

================Logstash File beat Input: =============================
input {
beats {
port => 5044
ssl => true
ssl_certificate_authorities => ["/etc/logstash/conf.d/xxx_xxx_G2_Intermediate_CA.cer", "/etc/logstash/conf.d/xxx_xxx_G2_Root_CA.cer"]
ssl_certificate => "/etc/logstash/conf.d/openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com.cer"
ssl_key => "/etc/logstash/conf.d/openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com-pkcs8.key"
ssl_verify_mode => "peer"
}
}

[stephenb (Stephen Brown)]

Well the first things I see is you have the logstash output of the filebeats file configured to

openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com:443

But it should be configured to port 5044 Because you have Logstash beats input listening on 5044. (Normal beats listening port for Logstash)

openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.com:5044

That would be the first thing I would fix.

Then report back.

[Rohan-boogeyman (Rohan)]

Thanks for your prompt response @stephenb but i tried that too and i was getting an error message:

2021-03-07T00:30:40.906-0500 ERROR [publisher_pipeline_output] pipeline
/output.go:154 Failed to connect to backoff(async(tcp://openshift-logstash-xxx
-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx:5044)): dial tcp 10.240.196.184
:5044: connectex: No connection could be made because the target machine activel
y refused it.

Now this port worked fine when we had no certs applied - issues started happening only when we took the approach of having ssl route.
Also to add earlier - i was using Logstash node ulvocpd082.xxx.xxx.com and its Service Port 32460 to connect with Filebeats and since this idea of using SSL certs was introduced i was recommended to use "complete route" along with port "443" and i started experiencing issues.

[stephenb (Stephen Brown)]

Right now you need to get your ports, ssl and certs settled / fixed.

If you listen on 5044 then you send on 5044, if you want to use 443 , then you need to send on 443 and listen on Logstash beats input on 443, which is fine. Don't mix and match.

2nd / Next You have tried to set up the most strictest SSL policies at the very beginning.

First I would try less strict and then iterate to strict.

Example on the file beat log stash output section

I would look at these settings.

First I would try this

ssl.verfication_mode: none

Also in the Logstash beats input you set the most strict ssl_verify_mode policy which will be fine but I think you need to start and get the connectivity and then fix your SSL.

Perhaps start with and then refine

ssl_verify_mode => none

Note this is not for production but should test whether the connectivity works.

then if all that works you can start to try to use the stricter SSL policies and verification modes

[Rohan-boogeyman (Rohan)]

@stephenb

ssl.verfication_mode: none

Hi @stephenb,

fixed both and i am still getting:
2021-03-07T01:32:00.410-0500 ERROR [publisher_pipeline_output] pipeline /output.go:154 Failed to connect to backoff(async(tcp://openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx:443)): EOF

===================Filebeat YML=======================================
hosts: ["openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx:443"]
ssl.certificate_authorities: ["C:\certs\openshift logstash\xxx_xxx_G2_Intermediate_CA.cer", "C:\certs\openshift logstash\xxx_xxx_G2_Root_CA.cer"]
ssl.certificate: "C:\certs\openshift logstash\openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx.cer"
ssl.key: "C:\certs\openshift logstash\openshift-logstash-xxx-xxx-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx.key"
ssl.enabled: true
ssl.verfication_mode: none

======================= Logstash Defaultpipeline.yml====================
input {
beats {
port => 443
ssl => true
ssl_certificate_authorities => ["/etc/logstash/conf.d/xxx_xxx_G2_Intermediate_CA.cer", "/etc/logstash/conf.d/xxx_xxx_G2_Root_CA.cer"]
ssl_certificate => "/etc/logstash/conf.d/openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx.cer"
ssl_key => "/etc/logstash/conf.d/openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx-pkcs8.key"
ssl_verify_mode => "none"
}
}

[stephenb (Stephen Brown)]

At this point I suspect you have a connectiivty issues between the two hosts on those ports, a Firewall, routing, proxy, nat or something. I notice you are running in open shift, does it only allow certain ports? Does ssl termination happen elsewhere? I am not an open shift expert so I will not be able to help much there.

An easy way to test is start Logstash with the ssl disabled.

ssl => false

Or comment out all the ssl settings

Then try to telnet from the filebeat host to the logstash host it should connect, if it does not you have a connectivity issue. I I'm not an open shift expert so I will not be able to help you with that.

From the filebeat server / container

telnet openshift-logstash-xxx-dev-cl1-xxx-xxx.apps.c1-ocp-dc1.xxx.xxx.xxx 443

[stephenb (Stephen Brown)]

Also how are you starting logstash can you see in the logstash startup logs that Logstash is actually starting , starting the right pipeline and listening on the right port?.. you should be able to see all that in the logs

[Rohan-boogeyman (Rohan)]

@stephenb - So this issue got resolved - had to involve an OCP guy - so it appears Host configuration in filebeat Yml file was supposed to point to 443 - and Filebeat Input within Logstash was 5044 - That part was correct.
Issue was OCP was configured as "NodePort" which had to be changed to "ClusterIP" to allow full route be used to send logs from Beats to Logstash.

[Rohan-boogeyman (Rohan)]

Now Logs are flowing fine - however i am getting a new error within OCP that has to do with "Connection reset by peer" in logstash - for which i did apply suggestion [Solved] Filebeat -> Logstash : connection reset by peer
even though I increased client_inactivity_timeout to 60k within Logstash - still continue to get this error.
Logs are flowing just fine from Filebeat to Kibana but this error is something i'd like to have fixed to ensure a clean implementation of certs.

[stephenb (Stephen Brown)]

Hi @Rohan-boogeyman

That is good news, glad things are working better!

Interesting configuration / network topology.

With respect to your other issue connection reset peer, in light of your moderately complex architecture and network topology, running logstash in openshift, it is possible there may be a component (A switch, router, FW, Load Balancer) in between filebeat and logstash which may be terminating the connection, I have seen this before.

Can you ask your OCP person if long lived connections are allowed? Or is there anything that mught be affecting long lived connections.