# Logstash not able to start with multiple pipelines **URL:** \ **Category:** Logstash\ **Tags:** docker\ **Created:** [October 29, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-start-with-multiple-pipelines/288029 "2021-10-29T18:04:27Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![dyadav](https://avatars.discourse-cdn.com/v4/letter/d/9fc348/32.png) [@dyadav](https://discuss.elastic.co/u/dyadav)\ **Post date:** [October 29, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-start-with-multiple-pipelines/288029/1 "2021-10-29T18:04:28Z")
I have setup logstash on EKS in AWS. Now, I am trying to setup multiple pipelines in it but it is not able to pick up my pipeline or just picks the main pipeline. ## Below are my configurations: ConfigMap apiVersion: v1 kind: ConfigMap metadata: name: logstash-configmap namespace: kube-system data: logstash.yml: | http.host: "0.0.0.0" log.level: trace pipelines.yml: | - pipeline.id: msst path.config: "/usr/share/logstash/persistent/config-msst.conf" ## apiVersion: apps/v1 kind: Deployment metadata: name: logstash namespace: kube-system spec: replicas: 1 selector: matchLabels: app: logstash template: metadata: labels: app: logstash spec: containers: - name: logstash image: [docker.elastic.co/logstash/logstash:7.12.1](http://docker.elastic.co/logstash/logstash:7.12.1) ports: - containerPort: 5044 volumeMounts: - name: config-volume mountPath: /usr/share/logstash/config - name: logstash-pipeline-volume mountPath: /usr/share/logstash/config - name: persistent-storage-db2-driver mountPath: /usr/share/logstash/persistent volumes: - name: config-volume configMap: name: logstash-configmap items: - key: logstash.yml path: logstash.yml - name: logstash-pipeline-volume configMap: name: logstash-configmap items: - key: pipelines.yml path: pipelines.yml - name: persistent-storage-db2-driver persistentVolumeClaim: claimName: efs-logstash-db2-driver-claim Logs: Using bundled JDK: /usr/share/logstash/jdk warning: no jvm.options file found Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2021-10-29 04:33:14.232 [main] runner - Starting Logstash {"logstash.version"=\>"7.12.1", "jruby.version"=\>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +jit [linux-x86\_64]"} [DEBUG] 2021-10-29 04:33:14.236 [main] scaffold - Found module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [DEBUG] 2021-10-29 04:33:14.237 [main] registry - Adding plugin to the registry {:name=\>"fb\_apache", :type=\>:modules, :class=\>#\} [DEBUG] 2021-10-29 04:33:14.237 [main] scaffold - Found module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} [DEBUG] 2021-10-29 04:33:14.238 [main] registry - Adding plugin to the registry {:name=\>"netflow", :type=\>:modules, :class=\>#\} [INFO] 2021-10-29 04:33:14.241 [main] writabledirectory - Creating directory {:setting=\>"path.queue", :path=\>"/usr/share/logstash/data/queue"} [INFO] 2021-10-29 04:33:14.242 [main] writabledirectory - Creating directory {:setting=\>"path.dead\_letter\_queue", :path=\>"/usr/share/logstash/data/dead\_letter\_queue"} [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - -------- Logstash Settings (\* means modified) --------- [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - node.name: "logstash-8b69574c4-kskjp" [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - path.data: "/usr/share/logstash/data" [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules.cli: \ [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules: [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules\_list: [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules\_variable\_list: [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules\_setup: false [DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - config.test\_and\_exit: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.reload.automatic: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.reload.interval: #Java::OrgLogstashUtil::TimeValue:0x3f3b50df [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.support\_escapes: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.field\_reference.parser: "STRICT" [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - metric.collect: true [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.id: "main" [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.system: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.workers: 1 [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.batch.size: 125 [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.batch.delay: 50 [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.unsafe\_shutdown: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.java\_execution: true [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.reloadable: true [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.plugin\_classloaders: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.separate\_logs: false [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.ordered: "auto" [DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.ecs\_compatibility: "disabled" [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - path.plugins: [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - config.debug: false [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - \*log.level: "trace" (default: "info") [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - version: false [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - help: false [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - log.format: "plain" [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.enabled: true [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - \*http.host: "0.0.0.0" (default: "127.0.0.1") [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.port: 9600..9700 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.environment: "production" [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.type: "memory" [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.drain: false [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.page\_capacity: 67108864 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.max\_bytes: 1073741824 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.max\_events: 0 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.acks: 1024 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.writes: 1024 [DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.interval: 1000 [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - queue.checkpoint.retry: false [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead\_letter\_queue.enable: false [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead\_letter\_queue.max\_bytes: 1073741824 [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead\_letter\_queue.flush\_interval: 5000 [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - slowlog.threshold.warn: #Java::OrgLogstashUtil::TimeValue:0x2174acc0 [DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - slowlog.threshold.info: #Java::OrgLogstashUtil::TimeValue:0x4e5361df [DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - slowlog.threshold.debug: #Java::OrgLogstashUtil::TimeValue:0x6b41df80 [DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - slowlog.threshold.trace: #Java::OrgLogstashUtil::TimeValue:0x35537fe [DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore" [DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore" [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.queue: "/usr/share/logstash/data/queue" [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.dead\_letter\_queue: "/usr/share/logstash/data/dead\_letter\_queue" [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config" [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs" [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.enabled: false [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.logstash.poll\_interval: #Java::OrgLogstashUtil::TimeValue:0xb41c538 [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"] [DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash\_system" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["[https://localhost:9200](https://localhost:9200)"] [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification\_mode: "certificate" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.enabled: false [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["[http://localhost:9200](http://localhost:9200)"] [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.interval: #Java::OrgLogstashUtil::TimeValue:0x2cb34e1e [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.timeout\_interval: #Java::OrgLogstashUtil::TimeValue:0x1de306c9 [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash\_system" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification\_mode: "certificate" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.enabled: false [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.elasticsearch.hosts: ["[http://localhost:9200](http://localhost:9200)"] [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.collection.interval: #Java::OrgLogstashUtil::TimeValue:0xcff1a53 [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.collection.timeout\_interval: #Java::OrgLogstashUtil::TimeValue:0x12cea099 [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.elasticsearch.username: "logstash\_system" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.elasticsearch.ssl.verification\_mode: "certificate" [DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.elasticsearch.sniffing: false [DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - monitoring.collection.pipeline.details.enabled: true [DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - monitoring.collection.config.enabled: true [DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - node.uuid: "" [DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - --------------- Logstash Settings ------------------- [DEBUG] 2021-10-29 04:33:14.414 [LogStash::Runner] multilocal - Reading pipeline configurations from YAML {:location=\>"/usr/share/logstash/config/pipelines.yml"} ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml usage: bin/logstash -f CONFIG\_PATH [-t] [-r] [-w COUNT] [-l LOG] bin/logstash --modules MODULE\_NAME [-M "MODULE\_NAME.var.PLUGIN\_TYPE.PLUGIN\_NAME.VARIABLE\_NAME=VALUE"] [-t] [-w COUNT] [-l LOG] bin/logstash -e CONFIG\_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG] bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace] bin/logstash -V [--log.level fatal|error|warn|info|debug|trace] bin/logstash --help [FATAL] 2021-10-29 04:33:14.421 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?] at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 26, 2021, 6:04pm UTC](https://discuss.elastic.co/t/logstash-not-able-to-start-with-multiple-pipelines/288029/2 "2021-11-26T18:04:40Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.