I have setup logstash on EKS in AWS. Now, I am trying to setup multiple pipelines in it but it is not able to pick up my pipeline or just picks the main pipeline.
Below are my configurations:
ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash-configmap
namespace: kube-system
data:
logstash.yml: |
http.host: "0.0.0.0"
log.level: trace
pipelines.yml: |
- pipeline.id: msst
path.config: "/usr/share/logstash/persistent/config-msst.conf"
apiVersion: apps/v1
kind: Deployment
metadata:
name: logstash
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: logstash
template:
metadata:
labels:
app: logstash
spec:
containers:
- name: logstash
image: docker.elastic.co/logstash/logstash:7.12.1
ports:
- containerPort: 5044
volumeMounts:
- name: config-volume
mountPath: /usr/share/logstash/config
- name: logstash-pipeline-volume
mountPath: /usr/share/logstash/config
- name: persistent-storage-db2-driver
mountPath: /usr/share/logstash/persistent
volumes:
- name: config-volume
configMap:
name: logstash-configmap
items:
- key: logstash.yml
path: logstash.yml
- name: logstash-pipeline-volume
configMap:
name: logstash-configmap
items:
- key: pipelines.yml
path: pipelines.yml
- name: persistent-storage-db2-driver
persistentVolumeClaim:
claimName: efs-logstash-db2-driver-claim
Logs:
Using bundled JDK: /usr/share/logstash/jdk
warning: no jvm.options file found
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2021-10-29 04:33:14.232 [main] runner - Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +jit [linux-x86_64]"}
[DEBUG] 2021-10-29 04:33:14.236 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2021-10-29 04:33:14.237 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x688c1b2a @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2021-10-29 04:33:14.237 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2021-10-29 04:33:14.238 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x2e15ad9b @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[INFO ] 2021-10-29 04:33:14.241 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2021-10-29 04:33:14.242 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) ---------
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - node.name: "logstash-8b69574c4-kskjp"
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - path.data: "/usr/share/logstash/data"
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 >
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules:
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules_list:
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules_variable_list:
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - modules_setup: false
[DEBUG] 2021-10-29 04:33:14.398 [LogStash::Runner] runner - config.test_and_exit: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.reload.automatic: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.reload.interval: #Java::OrgLogstashUtil::TimeValue:0x3f3b50df
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.support_escapes: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - config.field_reference.parser: "STRICT"
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - metric.collect: true
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.id: "main"
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.system: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.workers: 1
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.batch.size: 125
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.batch.delay: 50
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.java_execution: true
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.reloadable: true
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.plugin_classloaders: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.separate_logs: false
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.ordered: "auto"
[DEBUG] 2021-10-29 04:33:14.399 [LogStash::Runner] runner - pipeline.ecs_compatibility: "disabled"
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - path.plugins:
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - config.debug: false
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - *log.level: "trace" (default: "info")
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - version: false
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - help: false
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - log.format: "plain"
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.enabled: true
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - *http.host: "0.0.0.0" (default: "127.0.0.1")
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.port: 9600..9700
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - http.environment: "production"
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.type: "memory"
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.drain: false
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.page_capacity: 67108864
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.max_bytes: 1073741824
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.max_events: 0
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.acks: 1024
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.writes: 1024
[DEBUG] 2021-10-29 04:33:14.400 [LogStash::Runner] runner - queue.checkpoint.interval: 1000
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - queue.checkpoint.retry: false
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead_letter_queue.enable: false
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - dead_letter_queue.flush_interval: 5000
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - slowlog.threshold.warn: #Java::OrgLogstashUtil::TimeValue:0x2174acc0
[DEBUG] 2021-10-29 04:33:14.401 [LogStash::Runner] runner - slowlog.threshold.info: #Java::OrgLogstashUtil::TimeValue:0x4e5361df
[DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - slowlog.threshold.debug: #Java::OrgLogstashUtil::TimeValue:0x6b41df80
[DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - slowlog.threshold.trace: #Java::OrgLogstashUtil::TimeValue:0x35537fe
[DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[DEBUG] 2021-10-29 04:33:14.409 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore"
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.queue: "/usr/share/logstash/data/queue"
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.dead_letter_queue: "/usr/share/logstash/data/dead_letter_queue"
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - path.logs: "/usr/share/logstash/logs"
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.enabled: false
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: #Java::OrgLogstashUtil::TimeValue:0xb41c538
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"]
[DEBUG] 2021-10-29 04:33:14.410 [LogStash::Runner] runner - xpack.management.Elasticsearch.username: "logstash_system"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.Elasticsearch.hosts: ["https://localhost:9200"]
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.Elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.management.Elasticsearch.sniffing: false
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.Elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.interval: #Java::OrgLogstashUtil::TimeValue:0x2cb34e1e
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: #Java::OrgLogstashUtil::TimeValue:0x1de306c9
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.Elasticsearch.username: "logstash_system"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.Elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.Elasticsearch.sniffing: false
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.enabled: false
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.Elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.collection.interval: #Java::OrgLogstashUtil::TimeValue:0xcff1a53
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.collection.timeout_interval: #Java::OrgLogstashUtil::TimeValue:0x12cea099
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.Elasticsearch.username: "logstash_system"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.Elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2021-10-29 04:33:14.411 [LogStash::Runner] runner - monitoring.Elasticsearch.sniffing: false
[DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - monitoring.collection.config.enabled: true
[DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - node.uuid: ""
[DEBUG] 2021-10-29 04:33:14.412 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[DEBUG] 2021-10-29 04:33:14.414 [LogStash::Runner] multilocal - Reading pipeline configurations from YAML {:location=>"/usr/share/logstash/config/pipelines.yml"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[FATAL] 2021-10-29 04:33:14.421 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]