Logstash api endpoints are not accessible outside the logstash container

We deployed the ELK stack in kubernetes cluster and everything is working fine. Now we are trying to monitor logstash from kibana stack monitoring using metricbeat. We could see that logstash section is not visible in stack monitoring. Later identified that the api endpoints of logstash were not accessible from outside the logstash container and might be the reason why metricbeat is not able to get the monitoring data from logstash.

We could see the below errors in nginx controller logs.

> 2021/09/14 03:17:00 [error] 18575#18575: *162502431 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 192.168.3.1, server: logstash.ek01t.kub.rsascandi.com, request: "GET / HTTP/1.1", upstream: "https://192.168.3.85:9600/", host: "logstash.ek01t.kub.rsascandi.com:443"
> 2021/09/14 03:17:00 [error] 18575#18575: *162502431 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 192.168.3.1, server: logstash.ek01t.kub.rsascandi.com, request: "GET / HTTP/1.1", upstream: "https://192.168.3.85:9600/", host: "logstash.ek01t.kub.rsascandi.com:443"
> 2021/09/14 03:17:00 [error] 18575#18575: *162502431 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 192.168.3.1, server: logstash.ek01t.kub.rsascandi.com, request: "GET / HTTP/1.1", upstream: "https://192.168.3.85:9600/", host: "logstash.ek01t.kub.rsascandi.com:443"
> 192.168.3.1 - elastic [14/Sep/2021:03:17:00 +0000] "GET / HTTP/1.1" 502 150 "-" "Go-http-client/1.1" 172 0.003 [default-logstash-logstash-9600] [] 192.168.3.85:9600, 192.168.3.85:9600, 192.168.3.85:9600 0, 0, 0 0.000, 0.000, 0.000 502, 502, 502 8b623323fe0ddab50d582c7bba420eb9

Also we could see the below erros in metricbeat logs

2021-09-14T03:19:38.358Z ERROR [logstash.node_stats] node_stats/node_stats.go:73 HTTP error 502 in : 502 Bad Gateway

Please find the logs of logstash pod

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2021-09-13T18:03:11,832][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2021-09-13T18:03:11,839][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.0", "jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[2021-09-13T18:03:11,931][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["onlinese*"], :poll_interval=>"TimeValue{duration=5, timeUnit=SECONDS}ns"}
[2021-09-13T18:03:12,445][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-13T18:03:13,127][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@elasticsearch-master:9200/]}}
[2021-09-13T18:03:13,819][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch-master:9200/"}
[2021-09-13T18:03:13,913][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.13.0) {:es_version=>7}
[2021-09-13T18:03:13,915][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2021-09-13T18:03:14,040][INFO ][logstash.configmanagement.elasticsearchsource] Configuration Management License OK
[2021-09-13T18:03:15,552][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-13T18:03:15,640][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@elasticsearch-master:9200/]}}
[2021-09-13T18:03:15,731][WARN ][logstash.configmanagement.elasticsearchsource] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch-master:9200/"}
[2021-09-13T18:03:15,739][INFO ][logstash.configmanagement.elasticsearchsource] Elasticsearch version determined (7.13.0) {:es_version=>7}
[2021-09-13T18:03:15,810][WARN ][logstash.configmanagement.elasticsearchsource] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2021-09-13T18:03:16,012][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-09-13T18:03:17,215][INFO ][org.reflections.Reflections] Reflections took 81 ms to scan 1 urls, producing 24 keys and 48 values
[2021-09-13T18:03:18,118][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2021-09-13T18:03:18,231][INFO ][logstash.outputs.elasticsearch][onlinese-test] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://elasticsearch-master:9200"]}
[2021-09-13T18:03:18,236][WARN ][logstash.outputs.elasticsearch][onlinese-test] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2021-09-13T18:03:18,314][INFO ][logstash.outputs.elasticsearch][onlinese-test] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://elastic:xxxxxx@elasticsearch-master:9200/]}}
[2021-09-13T18:03:18,348][WARN ][logstash.outputs.elasticsearch][onlinese-test] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch-master:9200/"}
[2021-09-13T18:03:18,356][INFO ][logstash.outputs.elasticsearch][onlinese-test] Elasticsearch version determined (7.13.0) {:es_version=>7}
[2021-09-13T18:03:18,411][WARN ][logstash.outputs.elasticsearch][onlinese-test] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2021-09-13T18:03:18,518][WARN ][logstash.javapipeline ][onlinese-test] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-09-13T18:03:18,622][INFO ][logstash.outputs.elasticsearch][onlinese-test] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2021-09-13T18:03:18,711][INFO ][logstash.javapipeline ][onlinese-test] Starting pipeline {:pipeline_id=>"onlinese-test", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["central pipeline management"], :thread=>"#<Thread:0x1864ee17 run>"}
[2021-09-13T18:03:20,029][INFO ][logstash.javapipeline ][onlinese-test] Pipeline Java execution initialization time {"seconds"=>1.31}
[2021-09-13T18:03:20,357][INFO ][logstash.javapipeline ][onlinese-test] Pipeline started {"pipeline.id"=>"onlinese-test"}
[2021-09-13T18:03:20,416][INFO ][logstash.inputs.http ][onlinese-test][6b090a7ef4bbb165f4afc92e852f7128516586db6d70e4e2ab464251e82778b6] Starting http input listener {:address=>"0.0.0.0:5044", :ssl=>"false"}
[2021-09-13T18:03:20,442][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:"onlinese-test"], :non_running_pipelines=>}


please find the logstash service

> 
> Name:              logstash-logstash
> Namespace:         default
> Labels:            app=logstash-logstash
>                    app.kubernetes.io/managed-by=Helm
>                    chart=logstash
>                    heritage=Helm
>                    release=logstash
> Annotations:       meta.helm.sh/release-name: logstash
>                    meta.helm.sh/release-namespace: default
> Selector:          app=logstash-logstash,chart=logstash,release=logstash
> Type:              ClusterIP
> IP:                10.99.51.20
> Port:              api  9600/TCP
> TargetPort:        9600/TCP
> Endpoints:         192.168.3.85:9600
> Session Affinity:  None
> Events:            <none>

please find the ingress configured for logstash

> 
> λ kubectl describe ingress logstash-logstash
> Name:             logstash-logstash
> Namespace:        default
> Address:          10.206.16.5
> Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
> TLS:
>   elastic-tls terminates logstash.ek01t.kub.rsascandi.com
> Rules:
>   Host                              Path  Backends
>   ----                              ----  --------
>   logstash.ek01t.kub.rsascandi.com
>                                     /       logstash-logstash:9600 (192.168.3.85:9600)
>                                     /logs   logstash-logstash:8080 ()
> Annotations:                        ingress.kubernetes.io/ssl-passthrough: true
>                                     kubernetes.io/ingress.class: nginx
>                                     meta.helm.sh/release-name: logstash
>                                     meta.helm.sh/release-namespace: default
>                                     nginx.ingress.kubernetes.io/backend-protocol: HTTPS
>                                     nginx.ingress.kubernetes.io/secure-backends: true
> Events:                             <none>

Welcome to our community! :smiley:

Can you please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

Are you setting http.host to 0.0.0.0 ?

Logstash per default only binds the API endpoint to the localhost ip address, if you need to access this API from outside the container/server, you need to set http.host to 0.0.0.0.

I do not have experience with Helm charts, but looking on the github repository, the default values does not set http.host to 0.0.0.0.

Yes. I do set the http.host. I tried without spaces and with 1 and 2 spaces. But same result :cry:

logstashConfig:

logstash.yml: |

xpack.management.pipeline.id: ["onlinese*"]

xpack.management.enabled: true

xpack.management.elasticsearch.hosts: ["https://elasticsearch-master:9200"]

xpack.management.elasticsearch.username: '${ELASTICSEARCH_USERNAME}'

xpack.management.elasticsearch.password: '${ELASTICSEARCH_PASSWORD}'

xpack.management.elasticsearch.ssl.certificate_authority: /usr/share/logstash/config/certs/ca.crt

xpack.management.logstash.poll_interval: 5s

config.reload.automatic: true

http.host:  "0.0.0.0"