Hosting Logstash in Cloud

[aerstelle (Adam)]

I'm a ElasticSearch cloud user, and want to push my Java Spring Boot logs into elasticsearch.

The best solution that I have found so far is to use a logback-logstash-appender, and push the logs to logstash, which will push them into elasticsearch.

Assuming this is a good approach, I'm trying to host logstash in our Google Kubernetes Engine. I've been trying to use the logstash helm chart to set things up. With the helm chart, i have what I think is a pretty basic setup:

Helm Chart Values:

persistence:
  enabled: true

logstashConfig:
  logstash.yml: |
    http.host: 0.0.0.0
    xpack.monitoring.enabled: false

logstashPipeline:
  jsonlogs.conf: |
    input {
      tcp {
        port => 9601
        codec => json_lines
      }
    }
    output { stdout { } }

service:
  type: NodePort
  ports:
    - name: http
      port: 9600
      protocol: TCP
      targetPort: 9600
    - name: json-log-port
      port: 9601
      protocol: TCP
      targetPort: 9601

My Ingress looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: devops-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: logstash-ip-address
    networking.gke.io/managed-certificates: "logstash-certificate"
    ingress.kubernetes.io/add-base-url: "true"
    kubernetes.io/ingress.allow-http: "false"
spec:
  rules:
    - host: logstash.mydomain.com
      http:
        paths:
          - backend:
              serviceName: logstash-logstash
              servicePort: json-log-port

When I point the ingress to the "http" port, everything appears to work fine. I can hit the external endpoint with a browser and I get the logstash json saying things are working well.

When I point the ingress to the "json-log-port", the ingress tries to do the health checks and it doesn't work. Logstash STDOUT has a bunch of jsonparsefailures in the output (preceeded by a bunch of Unrecognized token for 'GET','Host','User','Connection')

I want to expose the pipeline so that I can push json log files. I know just a little bit about both logstash and kubernetes / gke, and would think what I want to do is basic....and am just struggling.

Has anyone else figured out how to host logstash on a cloud AND use it externally? Am I even on the right path or am I down a bad path.