Logstash failed to start with error "Logstash stopped processing because of an error: (SystemExit) exit"

Elastic Stack Logstash
1

I am an amateur to ELK and working on setup the logstash to elasticsearch (on elastic cloud). So ultimately my logs will be shipping from local logstash machine to Elastic Cloud. At some point I was managed to check my log via Kibana on Elastic Cloud. Yet, thing somehow goes wrong and logstash failed to start.

I checked some related error but no good news.
https://discuss.elastic.co/t/pipeline-yml-not-working/117120
Error showed that I may check on pipeline.yml , yet I never found any pipeline.yml in my environment. Is there any troubleshooting direction I may need to dive into?

One of the .conf files under conf.d:

   input {
      beats {
        port => "5044"
        tags => [ "beat" ]
      }
    }
    filter {
      if "beat" in [tags] {
        mutate {
              ##add_tag => [ "conf_file_9500"]
            }
      }
    }
    output {
      if "beat" in [tags] {
        elasticsearch {
    #      hosts => elasticsearch
          hosts => "******"
          user => "******"
          password => "******" 
          index => "logstash-beats-%{+YYYY.MM.dd}"
          template_name => "logstash-beats"
          template => "/beats-template.json"
          template_overwrite => true
        }
      }
    }

Logstash.yml:

#   * error
#   * warn
#   * info (default)
#   * debug
#   * trace
#
log.level: debug
# path.logs:
path.logs: /var/log/logstash
#
# ------------ Other Settings --------------
#
# Where to find custom plugins
# path.plugins: [\*]
# pipeline.workers: 2
# pipeline.workers: 2
# pipeline.workers: 2

Logstash

   [2019-03-26T07:57:05,118][DEBUG][logstash.runner] config.support_escapes: false
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] config.field_reference.parser: "COMPAT"
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] metric.collect: true
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.id: "main"
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.system: false
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.workers: 12
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.output.workers: 1
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.batch.size: 125
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.batch.delay: 50
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.unsafe_shutdown: false
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.java_execution: false
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.reloadable: true
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] path.plugins: []
    [2019-03-26T07:57:05,119][DEBUG][logstash.runner] config.debug: false
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] *log.level: "debug" (default: "info")
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] version: false
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] help: false
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] log.format: "plain"
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.host: "127.0.0.1"
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.port: 9600..9700
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.environment: "production"
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.type: "memory"
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.drain: false
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.page_capacity: 67108864
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.max_bytes: 1073741824
    [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.max_events: 0
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.acks: 1024
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.writes: 1024
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.interval: 1000
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] dead_letter_queue.enable: false
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] dead_letter_queue.max_bytes: 1073741824
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.warn: -1
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.info: -1
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.debug: -1
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.trace: -1
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] keystore.file: "/usr/share/logstash/config/logstash.keystore"
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] path.queue: "/usr/share/logstash/data/queue"
    [2019-03-26T07:57:05,121][DEBUG][logstash.runner] path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"
    [2019-03-26T07:57:05,122][DEBUG][logstash.runner] path."/usr/share/logstash/config"
    [2019-03-26T07:57:05,122][DEBUG][logstash.runner] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
    [2019-03-26T07:57:05,122][DEBUG][logstash.runner] --------------- Logstash Settings -------------------
    [2019-03-26T07:57:05,216][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/usr/share/logstash/config/pipelines.yml"}
    [2019-03-26T07:57:05,279][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
    root@socadmin-virtual-machine:/etc/logstash# Logstash stopped processing because of an error: (SystemExit) exitLogstash stopped processing because of an error: (SystemExit) exit
2

It says it is reading /usr/share/logstash/config/pipelines.yml. What is in that file?

3

The directory "/usr/share/logstash" doesn't exist, so there isn't any pipeline.yml at all.

4

I modified Logstash.yml config file and now it works fine. Thanks guys!

5

Benjen, may I know what changes did you do in Logstash.yml file?

Thanks

6

I suspected the yml file was crashed and simply replaced the file content with a freshly installed one. Then everything backed to the track.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.