should I reinstall logstash 8.8?
als oI find:
"
Elasticsearch 8.0 does not require Logstash 8.0. An Elasticsearch 8.0 cluster will happily receive data from earlier versions of Logstash via the default HTTP communication layer. This provides some flexibility to decide when to upgrade Logstash relative to an Elasticsearch upgrade. It may or may not be convenient for you to upgrade them together, and it is not required to be done at the same time as long as Elasticsearch is upgraded first.
"
I noticed now at starting I get below warning:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
as I specified logstash.yml exists at /etc/logstash directly not at $LS_HOME/config and actually there is a folder called conf.d but there is no config folder as per Warning message.
should I move the yml file into conf.d with logstash .conf or create a new folder called config and move it there?
I have changed the yml location by creating config folder but still getting same WARNING. also I noticed that both paths are expected (includedc /etc/logstash which includes the file from the beginning). Is it a permission issue? so why I get same warning even I run the command with sudo?
yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo ./logstash -f /etc/logstash/conf.d/logstash.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
I have created a config folder under:
/usr/share/logstash
and moved logstash.yml there .. when starting I got below error:
Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable.
[FATAL] 2023-05-23 13:29:18.821 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
I fixed by granting permission to said folder:
yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo chmod 777 -R /var/lib/logstash
when starting again .. no warning for config folder .. it seems starting writing and debugging all and I can see stdout but yet no index is created at Elasticsearch:
yasser@Yasser-Dell:/usr/share/logstash/bin$ curl -XGET http://localhost:9200/_cat/indices
yasser@Yasser-Dell:/usr/share/logstash/bin$ ./logstash -f /etc/logstash/conf.d/logstash.conf
Using JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-05-23 13:29:18.807 [main] runner - The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[INFO ] 2023-05-23 13:29:18.810 [main] runner - Starting Logstash {"logstash.version"=>"7.17.10", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.6+10-Ubuntu-0ubuntu120.04.1 on 17.0.6+10-Ubuntu-0ubuntu120.04.1 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-23 13:29:18.811 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
[DEBUG] 2023-05-23 13:29:18.813 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2023-05-23 13:29:18.814 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x3109f8ae @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:29:18.815 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2023-05-23 13:29:18.815 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x58649b7e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable.
[FATAL] 2023-05-23 13:29:18.821 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo chmod 777 -R /var/lib/logstash
yasser@Yasser-Dell:/usr/share/logstash/bin$ ./logstash -f /etc/logstash/conf.d/logstash.conf
Using JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-05-23 13:30:23.385 [main] runner - The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[INFO ] 2023-05-23 13:30:23.388 [main] runner - Starting Logstash {"logstash.version"=>"7.17.10", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.6+10-Ubuntu-0ubuntu120.04.1 on 17.0.6+10-Ubuntu-0ubuntu120.04.1 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-23 13:30:23.389 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
[DEBUG] 2023-05-23 13:30:23.392 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2023-05-23 13:30:23.393 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x3109f8ae @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:30:23.393 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2023-05-23 13:30:23.394 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x58649b7e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:30:23.497 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) ---------
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - node.name: "Yasser-Dell"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - *path.config: "/etc/logstash/conf.d/logstash.conf"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 []>
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_list: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_variable_list: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_setup: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.test_and_exit: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.reload.automatic: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0xdeddccb>
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.support_escapes: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.field_reference.parser: "STRICT"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - metric.collect: true
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.id: "main"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.system: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.workers: 20
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.batch.size: 125
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.batch.delay: 50
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.java_execution: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.reloadable: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.plugin_classloaders: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.separate_logs: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.ordered: "auto"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.ecs_compatibility: "disabled"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - path.plugins: []
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - *config.debug: true (default: false)
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - *log.level: "debug" (default: "info")
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - version: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - help: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - enable-local-plugin-development: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - log.format: "plain"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.enabled: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.http.host: "127.0.0.1"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.http.port: 9600..9700
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.environment: "production"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.auth.type: "none"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.ssl.enabled: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.type: "memory"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.drain: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.page_capacity: 67108864
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.max_bytes: 1073741824
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.max_events: 0
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.acks: 1024
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.writes: 1024
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.interval: 1000
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.retry: true
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.enable: false
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.flush_interval: 5000
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x11d16eb8>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x367fd31a>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x59c68ed3>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x7459002d>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x157eac27>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x44cefd71>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.enabled: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x346c568d>
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x3c9b416a>
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.config.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - node.uuid: ""
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.enabled: false
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x4de37f33>
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"]
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[WARN ] 2023-05-23 13:30:23.515 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[DEBUG] 2023-05-23 13:30:23.533 [LogStash::Runner] agent - Initializing API WebServer {"api.http.host"=>"127.0.0.1", "api.http.port"=>9600..9700, "api.ssl.enabled"=>false, "api.auth.type"=>"none", "api.environment"=>"production"}
[DEBUG] 2023-05-23 13:30:23.548 [LogStash::Runner] service - [api-service] start
[DEBUG] 2023-05-23 13:30:23.584 [LogStash::Runner] agent - Setting up metric collection
[DEBUG] 2023-05-23 13:30:23.607 [LogStash::Runner] os - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.725 [LogStash::Runner] jvm - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.766 [LogStash::Runner] jvm - collector name {:name=>"G1 Young Generation"}
[DEBUG] 2023-05-23 13:30:23.768 [LogStash::Runner] jvm - collector name {:name=>"G1 Old Generation"}
[DEBUG] 2023-05-23 13:30:23.776 [LogStash::Runner] persistentqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.782 [LogStash::Runner] deadletterqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.995 [Agent thread] agent - Starting agent
[DEBUG] 2023-05-23 13:30:24.001 [Api Webserver] agent - Starting API WebServer (puma)
[DEBUG] 2023-05-23 13:30:24.027 [Agent thread] configpathloader - Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf.d/02-beats-input.conf", "/etc/logstash/conf.d/30-elasticsearch-output.conf", "/etc/logstash/conf.d/grok-example.conf", "/etc/logstash/conf.d/logstash-simple.conf", "/etc/logstash/conf.d/logstash.yml", "/etc/logstash/conf.d/logstash_apache.conf", "/etc/logstash/conf.d/logstash_grok.conf"]}
[DEBUG] 2023-05-23 13:30:24.028 [Agent thread] configpathloader - Reading config file {:config_file=>"/etc/logstash/conf.d/logstash.conf"}
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - -------- Logstash Config ---------
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - Config from source, source: LogStash::Config::Source::Local, pipeline_id:: main
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - Config string, protocol: file, id: /etc/logstash/conf.d/logstash.conf
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig -
input
{
file
{
path => "/home/yasser/balance_inquiry.log"
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_clean_after => "60 s"
}
}
output
{
elasticsearch
{
hosts => "localhost:9200"
index => "balance-inquiry"
}
stdout
{
codec => rubydebug
}
}
[DEBUG] 2023-05-23 13:30:24.051 [Agent thread] PipelineConfig - Merged config
[DEBUG] 2023-05-23 13:30:24.051 [Agent thread] PipelineConfig -
input
{
file
{
path => "/home/yasser/balance_inquiry.log"
start_position => "beginning"
sincedb_path => "/dev/null"
sincedb_clean_after => "60 s"
}
}
output
{
elasticsearch
{
hosts => "localhost:9200"
index => "balance-inquiry"
}
stdout
{
codec => rubydebug
}
}
[DEBUG] 2023-05-23 13:30:24.055 [Api Webserver] agent - Trying to start API WebServer {:port=>9600, :ssl_enabled=>false}[INFO ] 2023-05-23 13:30:24.074 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[DEBUG] 2023-05-23 13:30:24.085 [Agent thread] agent - Converging pipelines state {:actions_count=>1}
[DEBUG] 2023-05-23 13:30:24.092 [Converge PipelineAction::Create<main>] agent - Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[DEBUG] 2023-05-23 13:30:24.101 [Converge PipelineAction::Create<main>] SecretStoreFactory - Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[DEBUG] 2023-05-23 13:30:24.384 [Converge PipelineAction::Create<main>] Reflections - going to scan these urls:
jar:file:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar!/
[INFO ] 2023-05-23 13:30:24.432 [Converge PipelineAction::Create<main>] Reflections - Reflections took 47 ms to scan 1 urls, producing 119 keys and 419 values
[DEBUG] 2023-05-23 13:30:24.437 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.JsonDeserializer -> com.fasterxml.jackson.databind.deser.std.StdDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.deser.NullValueProvider -> com.fasterxml.jackson.databind.JsonDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> com.fasterxml.jackson.databind.deser.std.StdDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.AbstractFilterable$Builder -> org.apache.logging.log4j.core.appender.AbstractAppender$Builder
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.security.SecureClassLoader -> java.net.URLClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.ClassLoader -> java.security.SecureClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Closeable -> java.net.URLClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.AutoCloseable -> java.io.Closeable
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonFormatVisitors.JsonFormatVisitable -> com.fasterxml.jackson.databind.JsonSerializer
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilderFactory -> org.apache.logging.log4j.core.config.ConfigurationFactory[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.RubyObject -> org.jruby.RubyException
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.RubyBasicObject -> org.jruby.RubyObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.IRubyObject -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Comparable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.marshal.CoreObjectType -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.InstanceVariables -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.InternalVariables -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.ser.std.StdSerializer -> com.fasterxml.jackson.databind.ser.std.StdScalarSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.JsonSerializer -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonFormatVisitors.JsonFormatVisitable -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonschema.SchemaAware -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Throwable -> java.lang.Exception
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.lang.Throwable
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractList -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractCollection -> java.util.AbstractList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Collection -> java.util.AbstractCollection
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Iterable -> java.util.Collection
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.List -> java.util.AbstractList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Collection -> java.util.List
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.List -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.RandomAccess -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.AbstractFilterable -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.AbstractLifeCycle -> org.apache.logging.log4j.core.filter.AbstractFilterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle2 -> org.apache.logging.log4j.core.AbstractLifeCycle
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.LifeCycle2
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.Filterable -> org.apache.logging.log4j.core.filter.AbstractFilterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.filter.Filterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.Appender -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.Appender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.impl.LocationAware -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Exception -> java.io.IOException
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Exception -> java.lang.RuntimeException
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.message.MessageFactory -> org.apache.logging.log4j.message.MessageFactory2
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.AbstractLifeCycle -> org.apache.logging.log4j.core.filter.AbstractFilter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.Filter -> org.apache.logging.log4j.core.filter.AbstractFilter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.Filter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.constant.Constable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Comparable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LogEvent -> org.apache.logging.log4j.core.impl.Log4jLogEvent
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.apache.logging.log4j.core.LogEvent
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractMap -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Map -> java.util.AbstractMap
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Map -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.impl.LocationAwareLogEventFactory -> org.apache.logging.log4j.core.impl.LogEventFactory
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.apache.logging.log4j.message.Message
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.RuntimeException -> java.lang.IllegalArgumentException
[DEBUG] 2023-05-23 13:30:24.449 [Converge PipelineAction::Create<main>] SecretStoreFactory - Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[DEBUG] 2023-05-23 13:30:24.564 [Converge PipelineAction::Create<main>] registry - On demand adding plugin to the registry {:name=>"file", :type=>"input", :class=>LogStash::Inputs::File}
[DEBUG] 2023-05-23 13:30:24.589 [Converge PipelineAction::Create<main>] registry - On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
[DEBUG] 2023-05-23 13:30:24.645 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@id = "plain_fe49ab78-3618-43c6-83d9-f995b6fa59b8"
[DEBUG] 2023-05-23 13:30:24.645 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@enable_metric = true
[DEBUG] 2023-05-23 13:30:24.646 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@charset = "UTF-8"
[WARN ] 2023-05-23 13:30:24.650 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[DEBUG] 2023-05-23 13:30:24.674 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@start_position = "beginning"
[DEBUG] 2023-05-23 13:30:24.674 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@path = ["/home/yasser/balance_inquiry.log"]
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_clean_after = 60.0
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@id = "c22aed2b5b294f372c697137c06beb4a02f6fc8367040c748d70c7cfc569900d"
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_path = "/dev/null"
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@enable_metric = true
[DEBUG] 2023-05-23 13:30:24.678 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain id=>"plain_fe49ab78-3618-43c6-83d9-f995b6fa59b8", enable_metric=>true, charset=>"UTF-8">
[DEBUG] 2023-05-23 13:30:24.678 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@add_field = {}
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@stat_interval = 1.0
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@discover_interval = 15
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_write_interval = 15.0
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@delimiter = "\n"
type or paste code here
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create] file - config LogStash::Inputs::File/@delimiter = "\n"
Is this the last line? Should be more
sorry for many messages, but it seems issue is resolved!
just to conclude, what I did:
now the index is created and data is moved.
The question here:
yasser@Yasser-Dell:/usr/share/logstash/bin$ curl -XGET 127.0.0.1:9200/balance-inquiry/_search?pretty
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 62,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "balance-inquiry",
"_id" : "EgNLSIgB_xR7YnnXbwgI",
"_score" : 1.0,
"_source" : {
"message" : " customerType=1",
"host" : "Yasser-Dell",
"@version" : "1",
"path" : "/home/yasser/balance_inquiry.log",
"@timestamp" : "2023-05-23T11:08:02.061Z"
}
},
{
"_index" : "balance-inquiry",
"_id" : "FQNLSIgB_xR7YnnXbwgI",
"_score" : 1.0,
"_ignored" : [
"message.keyword"
],
"_source" : {
"message" : "INFO: final balanceInquiryReplyVO = [BalanceInquiryReplyVO [header=MessageHeaderVO [msgFormat=null, msgVersion=null, msgOption=null, requestorChannelID=null, replierChannelID=null, requestorChannelFunction=null, requestorUserID=null, lang=null, securityInfo=null, securityInfoKey=null, returnCode=0000, errorCode=null, errorMessage=0000, srDate=null, srRcvDate=null, srStatus=0, funcId=null, securityKey=null, requestorLanguage=null, overwriteOpenOrder=null, chargeable=null, serviceRequestId=null, backendChannel=null], lineNumber=966530507255, customerType=1, packageID=Mobily Prepaid MNP, balance=0.75, expirationDate=Sat Aug 12 00:00:00 GMT+03:00 2023, billedAmount=0.0, unbilledAmount=0.0, unalocatedAmount=0.0, inValue=0, accountNumber=null, outstandingAmount=0.0, amountDue=0.0, packageName=null, monthlySubscriptionFee=0.0, lastBillDate=null, lastUpdateTime=null, additionalRentalFees=0.0, renewalAmount=0.0, ftthRenewexpDate=null, tempFtthRenewexpDate=null, errorCode=null, errorMessage=null, ncraBalance=null, unAllocated=0.0]]",
Good job trudging through this!
And now like I said I think it was something simple that we all did not see.
Did you install as a package?
And then you were running from the command line.
And like I said I should have seen this...
If you install as a package then run as a command line you have to add additional parameters to tell it where the config is at.
--path.settings /etc/logstash
This tells logstash where logstash.yml and pipeline.yml are at.
So the command should be
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -r --path.settings /etc/logstash
If you look in the service startup file, you will see the command look like that.
If you had used tar.gz then your command would have worked.
I think if you set yourself all the way back and ran it like this it would work..
Also if you started it via the service it would work.
Now if you start logstash via the service /systemctl after you've moved everything you're going to run into another problem because they were looking /etc/logstash... And you will be out of sync and other frustrating situation may occur
What I would probably do is save off all your config files.... Completely uninstall the package and make sure it cleans up everything because now you moved files around.
Reinstall put all your correct config in the right place.
Try the command I showed above.
Then try starting as a service.
Apologies... Should have seen that
trace != debug. filewatch provides helpful logging only at trace level.
Thanks a lot @stephenb for clarifying the impact of installing as a package and running as a service. actually there is a limitation at WSL in tunning as a service (I run Elasticsearch as a service but neither kibana nor lostash). I have reinstalled all 3 products from tar file not debian but still have this shortage at kibana and logstash. I'll work on this any way.
only last question before closing this topic, I never edited pipelines.yml and I do not know where it should be located. is it impacted or created by logstash.conf or should it have different type of configuration?
I saw many videos talking about pipelines while they edit input and output at only logstash.conf. This is why it is confusing a little.
I suggest you review the directory layouts for logstash installations
In short, if you install as a tar.gz everything is located below the extracted directory.
If you install as a package, the configuration is in /etc/logstash and below and the binaries and logs etc. Where they should be.
And let's be really precise and clear again. I'm not as familiar with the Ubuntu on windows.
On Ubuntu you run as a service with systemctl , When I say service that is what I mean.
Not a window service.
Apologies, I don't know what that means...
So if you install by tar.gz should look like
hyperion:logstash-8.7.0 sbrown$ pwd
/Users/sbrown/workspace/elastic-install/8.7.0/logstash-8.7.0
hyperion:logstash-8.7.0 sbrown$ ls
CONTRIBUTORS NOTICE.TXT lib/ tools/
Gemfile bin/ logs/ vendor/
Gemfile.lock config/ logstash-core/ x-pack/
JDK_VERSION data/ logstash-core-plugin-api/
LICENSE.txt jdk.app/ modules/
hyperion:logstash-8.7.0 sbrown$ ls config/
jvm.options logstash-sample.conf pipelines.yml
log4j2.properties logstash.yml startup.options
hyperion:logstash-8.7.0 sbrown$
if you install as a package
[root@stephenb-centos-logstash ~]# cd /etc/logstash
[root@stephenb-centos-logstash logstash]# ls -lrt
total 40
-rw-r--r--. 1 root root 285 Feb 12 05:39 pipelines.yml
-rw-r--r--. 1 root root 342 Feb 12 05:39 logstash-sample.conf
-rw-r--r--. 1 root root 7437 Feb 12 05:39 log4j2.properties
-rw-r--r--. 1 root root 1833 Feb 12 05:39 jvm.options
-rw-r--r--. 1 root root 15011 Mar 10 02:12 logstash.yml
-rw-------. 1 root root 1764 Mar 10 02:48 startup.options
drwxr-xr-x. 2 root root 34 Mar 10 02:52 conf.d
[root@stephenb-centos-logstash logstash]#
you use pipelines.yml see here if you want to run independent pipelines otherwise logstash will concatenate everything in the conf.d into 1 giant pipeline...
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.