Output isolator pattern not working as expected when elastic is not reachable

Elastic Stack Logstash
1

Hi,

I have implemented output isolator pattern having with 4 pipelines, main, elastic, http and mongodb. Now when the connectivity of elastic is lost, it doesn't push events to http and mongodb also.

I have gone through similar issues posted in this forum but they are using persistent queues while i am not using.

deployment.yaml

# filepath: k8s/logstash/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: logstash
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: logstash
  template:
    metadata:
      labels:
        app: logstash
    spec:      
      containers:
      - name: logstash
        image: <logstash:8.15.5>
        env:
        - name: ELASTICSEARCH_HOSTS
          value: "elastic-aws-deployment:9200"
        - name: HTTP_ENDPOINT
          value: "http://webhook-server/webhook1"
        - name: MONGODB_URI
          value: "mongodb://mongodb.utils:27017/Logs"
        - name: LS_JAVA_OPTS
          value: "-Xmx2g -Xms2g"
        - name: LOG_LEVEL
          value: "debug"
        volumeMounts:
        - name: config-volume
          mountPath: /usr/share/logstash/config/pipeline
        - name: pipelines-yml
          mountPath: /usr/share/logstash/config/pipelines.yml
          subPath: pipelines.yml
        - name: plugins
          mountPath: /usr/share/logstash/plugins  
      volumes:
      - name: config-volume
        configMap:
          name: logstash-config
          items:
          - key: input.conf
            path: input.conf
          - key: output_elasticsearch.conf
            path: output_elasticsearch.conf
          - key: output_http.conf
            path: output_http.conf
          - key: output_mongo.conf
            path: output_mongo.conf
      - name: pipelines-yml
        configMap:
          name: logstash-config
          items:
          - key: pipelines.yml
            path: pipelines.yml
      - name: plugins
        emptyDir: {}

configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-config
data:
  input.conf: |
    input {
      beats {
        port => 5044
      }
    }
    filter {
      json {
        source => "message"
      }
      mutate {
        remove_field => ["[event][original]", "[tags]", "[agent][ephemeral_id]", "[agent][name]", "[agent][version]", "[ecs][version]"]
        add_field => { "[host-ip]" => "dummy" }
        convert => {
          "[duration-in-seconds]" => "string"
          "[message][duration-in-seconds]" => "string"
        }
      }
      date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
      geoip {
        source => "clientip"
        ecs_compatibility => disabled
      }

      if [destination-type] == "EVENTS" {
        mutate { add_field => { "[@metadata][indexPrefix]" => "events" } }
      }
    }
    output {
      pipeline {
        send_to => ["elastic_output", "mongo_output", "http_output"]
      }
    }


  output_elasticsearch.conf: |
    input {
      pipeline {
        address => "elastic_output"
      }
    }
    output {
      elasticsearch {
        hosts => ["${ELASTICSEARCH_HOSTS}"]
        index => "%{[@metadata][indexPrefix]}-%{+YYYY.MM}"
      }
    }

  output_http.conf: |
    input {
      pipeline {
        address => "http_output"
      }
    }
    output {
      http {
        url => "${HTTP_ENDPOINT}"
        http_method => "post"
        format => "json"
      }
    }

  output_mongo.conf: |
    input {
      pipeline {
        address => "mongo_output"
      }
    }
    output {
      mongodb {
        id => "mongo"
        collection => "logs"
        database => "Logs"
        uri => "${MONGODB_URI}"
      }
    }

  pipelines.yml: |
    - pipeline.id: main
      path.config: "/usr/share/logstash/config/pipeline/input.conf"
      pipeline.workers: 3
      pipeline.batch.size: 50
      pipeline.batch.delay: 10
    - pipeline.id: elastic_output
      path.config: "/usr/share/logstash/config/pipeline/output_elasticsearch.conf"
      pipeline.workers: 3
      pipeline.batch.size: 50
      pipeline.batch.delay: 10
    - pipeline.id: http_output
      pipeline.workers: 3
      pipeline.batch.size: 50
      pipeline.batch.delay: 10
      path.config: "/usr/share/logstash/config/pipeline/output_http.conf"
    - pipeline.id: mongo_output
      pipeline.workers: 3
      pipeline.batch.size: 50
      pipeline.batch.delay: 10
      path.config: "/usr/share/logstash/config/pipeline/output_mongo.conf"
2

Hello and welcome,

This is expected, the output isolator pattern can be a little misleading sometimes, as a blocked output can still block the other inputs.

In your case your first pipeline has an output that sends to 3 other pipelines using the pipeline-to-pipeline communicaton.

One of those pipelines has an Elasticsearch output, so if this output is not working, the logstash memory queue will start to fill up and once it is full it stops accepting new events, this will then propagate to the first pipeline, which will tell the beats input to backoff, this will affect all other outputs in this pipeline.

One alternative would be to use some udp input/output that would drop the evens instead of blocking it, but this also may not be what you want.

This example is described in this similar post.

3

Thanks a lot Leandro, this makes sense now. will look into the UDP option.