Hosting Logstash in Cloud

I'm a ElasticSearch cloud user, and want to push my Java Spring Boot logs into elasticsearch.

The best solution that I have found so far is to use a logback-logstash-appender, and push the logs to logstash, which will push them into elasticsearch.

Assuming this is a good approach, I'm trying to host logstash in our Google Kubernetes Engine. I've been trying to use the logstash helm chart to set things up. With the helm chart, i have what I think is a pretty basic setup:

Helm Chart Values:

  enabled: true

  logstash.yml: |
    xpack.monitoring.enabled: false

  jsonlogs.conf: |
    input {
      tcp {
        port => 9601
        codec => json_lines
    output { stdout { } }

  type: NodePort
    - name: http
      port: 9600
      protocol: TCP
      targetPort: 9600
    - name: json-log-port
      port: 9601
      protocol: TCP
      targetPort: 9601

My Ingress looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
  name: devops-ingress
  annotations: logstash-ip-address "logstash-certificate" "true" "false"
    - host:
          - backend:
              serviceName: logstash-logstash
              servicePort: json-log-port

When I point the ingress to the "http" port, everything appears to work fine. I can hit the external endpoint with a browser and I get the logstash json saying things are working well.

When I point the ingress to the "json-log-port", the ingress tries to do the health checks and it doesn't work. Logstash STDOUT has a bunch of jsonparsefailures in the output (preceeded by a bunch of Unrecognized token for 'GET','Host','User','Connection')

I want to expose the pipeline so that I can push json log files. I know just a little bit about both logstash and kubernetes / gke, and would think what I want to do is basic....and am just struggling.

Has anyone else figured out how to host logstash on a cloud AND use it externally? Am I even on the right path or am I down a bad path.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.