I am an amateur to ELK and working on setup the logstash to elasticsearch (on elastic cloud). So ultimately my logs will be shipping from local logstash machine to Elastic Cloud. At some point I was managed to check my log via Kibana on Elastic Cloud. Yet, thing somehow goes wrong and logstash failed to start.
I checked some related error but no good news.
https://discuss.elastic.co/t/pipeline-yml-not-working/117120
Error showed that I may check on pipeline.yml , yet I never found any pipeline.yml in my environment. Is there any troubleshooting direction I may need to dive into?
One of the .conf files under conf.d:
input { beats { port => "5044" tags => [ "beat" ] } } filter { if "beat" in [tags] { mutate { ##add_tag => [ "conf_file_9500"] } } } output { if "beat" in [tags] { elasticsearch { # hosts => elasticsearch hosts => "******" user => "******" password => "******" index => "logstash-beats-%{+YYYY.MM.dd}" template_name => "logstash-beats" template => "/beats-template.json" template_overwrite => true } } }
Logstash.yml:
# * error # * warn # * info (default) # * debug # * trace # log.level: debug # path.logs: path.logs: /var/log/logstash # # ------------ Other Settings -------------- # # Where to find custom plugins # path.plugins: [\*] # pipeline.workers: 2 # pipeline.workers: 2 # pipeline.workers: 2
Logstash
[2019-03-26T07:57:05,118][DEBUG][logstash.runner] config.support_escapes: false [2019-03-26T07:57:05,119][DEBUG][logstash.runner] config.field_reference.parser: "COMPAT" [2019-03-26T07:57:05,119][DEBUG][logstash.runner] metric.collect: true [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.id: "main" [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.system: false [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.workers: 12 [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.output.workers: 1 [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.batch.size: 125 [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.batch.delay: 50 [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.unsafe_shutdown: false [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.java_execution: false [2019-03-26T07:57:05,119][DEBUG][logstash.runner] pipeline.reloadable: true [2019-03-26T07:57:05,119][DEBUG][logstash.runner] path.plugins: [] [2019-03-26T07:57:05,119][DEBUG][logstash.runner] config.debug: false [2019-03-26T07:57:05,120][DEBUG][logstash.runner] *log.level: "debug" (default: "info") [2019-03-26T07:57:05,120][DEBUG][logstash.runner] version: false [2019-03-26T07:57:05,120][DEBUG][logstash.runner] help: false [2019-03-26T07:57:05,120][DEBUG][logstash.runner] log.format: "plain" [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.host: "127.0.0.1" [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.port: 9600..9700 [2019-03-26T07:57:05,120][DEBUG][logstash.runner] http.environment: "production" [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.type: "memory" [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.drain: false [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.page_capacity: 67108864 [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.max_bytes: 1073741824 [2019-03-26T07:57:05,120][DEBUG][logstash.runner] queue.max_events: 0 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.acks: 1024 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.writes: 1024 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] queue.checkpoint.interval: 1000 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] dead_letter_queue.enable: false [2019-03-26T07:57:05,121][DEBUG][logstash.runner] dead_letter_queue.max_bytes: 1073741824 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.warn: -1 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.info: -1 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.debug: -1 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] slowlog.threshold.trace: -1 [2019-03-26T07:57:05,121][DEBUG][logstash.runner] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [2019-03-26T07:57:05,121][DEBUG][logstash.runner] keystore.file: "/usr/share/logstash/config/logstash.keystore" [2019-03-26T07:57:05,121][DEBUG][logstash.runner] path.queue: "/usr/share/logstash/data/queue" [2019-03-26T07:57:05,121][DEBUG][logstash.runner] path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue" [2019-03-26T07:57:05,122][DEBUG][logstash.runner] path."/usr/share/logstash/config" [2019-03-26T07:57:05,122][DEBUG][logstash.runner] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs") [2019-03-26T07:57:05,122][DEBUG][logstash.runner] --------------- Logstash Settings ------------------- [2019-03-26T07:57:05,216][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/usr/share/logstash/config/pipelines.yml"} [2019-03-26T07:57:05,279][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit root@socadmin-virtual-machine:/etc/logstash# Logstash stopped processing because of an error: (SystemExit) exitLogstash stopped processing because of an error: (SystemExit) exit