Logstash pipeline terminates immediately after logstash container starts up

Hi everyone

I'm trying to set up logstash in a kubernetes cluster, but whenever the container starts the logs say it terminates the pipeline I have: [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}

My configuration files are as follows:
pipeline config

input {
      syslog {
        port => 5144
        host => "0.0.0.0"
      }
    }
    filter {
      grok {
        match => {
          "message" => [
            ",attack_type=\"%{DATA:attack_type}\"",
            ",blocking_exception_reason=\"%{DATA:blocking_exception_reason}\"",
            ",bot_signature_name=\"%{DATA:bot_signature_name}\"",
            ",bot_category=\"%{DATA:bot_category}\"",
            ",bot_anomalies=\"%{DATA:bot_anomalies}\"",
            ",client_class=\"%{DATA:client_class}\"",
            ",date_time=\"%{DATA:date_time}\"",
            ",dest_port=\"%{DATA:dest_port}\"",
            ",ip_client=\"%{DATA:ip_client}\"",
            ",is_truncated=\"%{DATA:is_truncated}\"",
            ",method=\"%{DATA:method}\"",
            ",policy_name=\"%{DATA:policy_name}\"",
            ",protocol=\"%{DATA:protocol}\"",
            ",request_status=\"%{DATA:request_status}\"",
            ",response_code=\"%{DATA:response_code}\"",
            ",severity=\"%{DATA:severity}\"",
            ",sig_cves=\"%{DATA:sig_cves}\"",
            ",sig_ids=\"%{DATA:sig_ids}\"",
            ",sig_names=\"%{DATA:sig_names}\"",
            ",sig_set_names=\"%{DATA:sig_set_names}\"",
            ",src_port=\"%{DATA:src_port}\"",
            ",sub_violations=\"%{DATA:sub_violations}\"",
            ",support_id=\"%{DATA:support_id}\"",
            ",unit_hostname=\"%{DATA:unit_hostname}\"",
            ",uri=\"%{DATA:uri}\"",
            ",violation_rating=\"%{DATA:violation_rating}\"",
            ",vs_name=\"%{DATA:vs_name}\"",
            ",x_forwarded_for_header_value=\"%{DATA:x_forwarded_for_header_value}\"",
            ",outcome=\"%{DATA:outcome}\"",
            ",outcome_reason=\"%{DATA:outcome_reason}\"",
            ",violations=\"%{DATA:violations}\"",
            ",violation_details=\"%{DATA:violation_details}\"",
            ",request=\"%{DATA:request}\""
          ]
        }
        break_on_match => false
      }
      mutate {
        split => { "attack_type" => "," }
        split => { "sig_ids" => "," }
        split => { "sig_names" => "," }
        split => { "sig_cves" => "," }
        split => { "staged_sig_ids" => "," }
        split => { "staged_sig_names" => "," }
        split => { "staged_sig_cves" => "," }
        split => { "sig_set_names" => "," }
        split => { "threat_campaign_names" => "," }
        split => { "staged_threat_campaign_names" => "," }
        split => { "violations" => "," }
        split => { "sub_violations" => "," }
      }
      if [x_forwarded_for_header_value] != "N/A" {
        mutate { add_field => { "source_host" => "%{x_forwarded_for_header_value}"}}
      } else {
        mutate { add_field => { "source_host" => "%{ip_client}"}}
      }
      geoip {
        source => "source_host"
      }
    }
    output {
      elasticsearch {
        hosts => "quickstart-es-http:9200"
        index => "waf-logs-%{+YYY.MM.dd}"
        user => "waf-logstash"
        password => "blahblah"
      }
    }

config file for logstash

http.host: "0.0.0.0"
http.port: "5144"
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: "quickstart-es-http:9200"
xpack.monitoring.elasticsearch.username: "waf-logstash"
xpack.monitoring.elasticsearch.password: "blahblah"

I pass the following arguments to the logstash command on start up
-f /usr/share/logstash/pipeline

I'm using 7.9.3 tagged image of logstash
I would greatly appreciate any help with this as I am a complete noob on ELK!

The full log output is as follows:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby15032451142012517880jopenssl.jar) to field java.security.MessageDigest.provider
WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-11-12 11:35:50.403 [main] runner - Starting Logstash {"logstash.version"=>"7.9.3", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10-LTS on 11.0.8+10-LTS +jit [linux-x86_64]"}
[INFO ] 2020-11-12 11:35:50.416 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-11-12 11:35:50.418 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2020-11-12 11:35:50.608 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-11-12 11:35:50.617 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"e2bc3af3-102b-4d42-9e1f-b8ecf260fe59", :path=>"/usr/share/logstash/data/uuid"}
[WARN ] 2020-11-12 11:35:50.691 [LogStash::Runner] pipelineregisterhook - Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
Please configure Metricbeat to monitor Logstash. Documentation can be found at: 
https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
[INFO ] 2020-11-12 11:35:51.072 [LogStash::Runner] licensereader - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:51.204 [LogStash::Runner] licensereader - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:51.364 [LogStash::Runner] licensereader - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Monitoring License OK
[INFO ] 2020-11-12 11:35:51.429 [LogStash::Runner] internalpipelinesource - Validated license for monitoring. Enabling monitoring pipeline.
[INFO ] 2020-11-12 11:35:51.442 [Agent thread] configpathloader - No config files found in path {:path=>"/usr/share/logstash/ /usr/share/logstash/pipeline"}
[INFO ] 2020-11-12 11:35:52.070 [Converge PipelineAction::Create<.monitoring-logstash>] Reflections - Reflections took 37 ms to scan 1 urls, producing 22 keys and 45 values 
[INFO ] 2020-11-12 11:35:52.141 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://waf-logstash:xxxxxx@quickstart-es-http:9200/]}}
[WARN ] 2020-11-12 11:35:52.151 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Restored connection to ES instance {:url=>"http://waf-logstash:xxxxxx@quickstart-es-http:9200/"}
[INFO ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - ES Output version determined {:es_version=>7}
[WARN ] 2020-11-12 11:35:52.158 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2020-11-12 11:35:52.186 [[.monitoring-logstash]-pipeline-manager] elasticsearchmonitoring - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["//quickstart-es-http:9200"]}
[WARN ] 2020-11-12 11:35:52.187 [[.monitoring-logstash]-pipeline-manager] javapipeline - 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[INFO ] 2020-11-12 11:35:52.210 [[.monitoring-logstash]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x393ee466@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:125 run>"}
[INFO ] 2020-11-12 11:35:52.961 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.75}
[INFO ] 2020-11-12 11:35:52.969 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2020-11-12 11:35:52.991 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:".monitoring-logstash"], :non_running_pipelines=>[]}
[INFO ] 2020-11-12 11:35:53.040 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>5144}
[INFO ] 2020-11-12 11:35:54.188 [[.monitoring-logstash]-pipeline-manager] javapipeline - Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[INFO ] 2020-11-12 11:35:55.028 [LogStash::Runner] runner - Logstash shut down.

Bumping this in hopes someone with even slightly more experience than me casts their gaze on this and decides to help

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.