i've managed to run elastic search with kibana and logstash with JDBC mssql driver for ingesting data.
Now i'm trying to run multiple pipelines but i keep getting an error.
ELK : 7.4.0 version
pipeline.yml :
# List of pipelines to be loaded by Logstash
#
# This document must be a list of dictionaries/hashes, where the keys/values are pipeline settings.
# Default values for omitted settings are read from the `logstash.yml` file.
# When declaring multiple pipelines, each MUST have its own `pipeline.id`.
#
# Example of two pipelines:
#
- pipeline.id: relation
#pipeline.workers: 1
path.config: "\Elastic_Stack\logstash-7.4.0\config\conf_relation.conf"
# pipeline.batch.size: 1
# config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }"
- pipeline.id: connectionpoint
# queue.type: persisted
path.config: "\Elastic_Stack\logstash-7.4.0\config\conf_connectionpoint.conf"
each seperate conf file runs with /bin/logstash -f conf_relation of /bin/logstash -f conf_connectionpoint.conf
directory of pipeline.yml : "C:\Elastic_Stack\logstash-7.4.0\config\pipelines.yml"
when running logstash i get this error:
C:\Elastic_Stack\logstash-7.4.0\bin>logstash
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/C:/Elastic_Stack/logstash-7.4.0/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.Console.cs
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to C:/Elastic_Stack/logstash-7.4.0/logs which is now configured via log4j2.properties
ERROR: Failed to read pipelines yaml file. Location: C:/Elastic_Stack/logstash-7.4.0/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[2019-12-04T19:45:42,649][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
just run your pipeline.yml with this configuration and see if you get it working.
your current pipeline.yml isn't specifying any workers for either of the pipelines so i think as far as logstash is concerned you've asked it to perform a task, but not given any resources to complete the task.
just remember... workers = CPU.... you want workers to match the workload expected.. i.e only a few light logs per worker is ok, but heavy logs take longer to parse and require more CPU to make it parse faster. Also using sincedb to keep timestamp access to log files in checks is a smart way to control log read times and also putting an interval per .conf file and try to stagger them... i.e B45.conf and B23.conf access logs on same client infrastructure, but one is every 23 minutes checking for changes, the other is every 45 minutes, but inside the .conf files we adjust and change and massage until we get what we need working the way we want it to for a while until its done and the logs coming into Elastic are parsed the way we want.
yes i cutted out the top part. But separately both files work with logstash
when running /bin/logstash -f conf_relation.conf or /bin/logstash -f conf_connectionpoint.conf
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.