Pipeline is running but index is not created at elasticsearch

Elastic Stack Logstash
21

should I reinstall logstash 8.8?
als oI find:

"
Elasticsearch 8.0 does not require Logstash 8.0. An Elasticsearch 8.0 cluster will happily receive data from earlier versions of Logstash via the default HTTP communication layer. This provides some flexibility to decide when to upgrade Logstash relative to an Elasticsearch upgrade. It may or may not be convenient for you to upgrade them together, and it is not required to be done at the same time as long as Elasticsearch is upgraded first.
"

22

I noticed now at starting I get below warning:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
as I specified logstash.yml exists at /etc/logstash directly not at $LS_HOME/config and actually there is a folder called conf.d but there is no config folder as per Warning message.
should I move the yml file into conf.d with logstash .conf or create a new folder called config and move it there?

23

I have changed the yml location by creating config folder but still getting same WARNING. also I noticed that both paths are expected (includedc /etc/logstash which includes the file from the beginning). Is it a permission issue? so why I get same warning even I run the command with sudo?

yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo ./logstash -f /etc/logstash/conf.d/logstash.conf

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
24

I have created a config folder under:
/usr/share/logstash
and moved logstash.yml there .. when starting I got below error:

Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable.
[FATAL] 2023-05-23 13:29:18.821 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

I fixed by granting permission to said folder:

yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo chmod 777 -R /var/lib/logstash

when starting again .. no warning for config folder .. it seems starting writing and debugging all and I can see stdout but yet no index is created at Elasticsearch:

yasser@Yasser-Dell:/usr/share/logstash/bin$ curl -XGET http://localhost:9200/_cat/indices

25 
yasser@Yasser-Dell:/usr/share/logstash/bin$ ./logstash -f /etc/logstash/conf.d/logstash.conf
Using JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-05-23 13:29:18.807 [main] runner - The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[INFO ] 2023-05-23 13:29:18.810 [main] runner - Starting Logstash {"logstash.version"=>"7.17.10", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.6+10-Ubuntu-0ubuntu120.04.1 on 17.0.6+10-Ubuntu-0ubuntu120.04.1 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-23 13:29:18.811 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
[DEBUG] 2023-05-23 13:29:18.813 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2023-05-23 13:29:18.814 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x3109f8ae @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:29:18.815 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2023-05-23 13:29:18.815 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x58649b7e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
Your settings are invalid. Reason: Path "/var/lib/logstash" must be a writable directory. It is not writable.
[FATAL] 2023-05-23 13:29:18.821 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
yasser@Yasser-Dell:/usr/share/logstash/bin$ sudo chmod 777 -R /var/lib/logstash
yasser@Yasser-Dell:/usr/share/logstash/bin$ ./logstash -f /etc/logstash/conf.d/logstash.conf
Using JAVA_HOME defined java: /usr/lib/jvm/java-17-openjdk-amd64
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2023-05-23 13:30:23.385 [main] runner - The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[INFO ] 2023-05-23 13:30:23.388 [main] runner - Starting Logstash {"logstash.version"=>"7.17.10", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 17.0.6+10-Ubuntu-0ubuntu120.04.1 on 17.0.6+10-Ubuntu-0ubuntu120.04.1 +indy +jit [linux-x86_64]"}
[INFO ] 2023-05-23 13:30:23.389 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED]
[DEBUG] 2023-05-23 13:30:23.392 [main] scaffold - Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[DEBUG] 2023-05-23 13:30:23.393 [main] registry - Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x3109f8ae @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:30:23.393 [main] scaffold - Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[DEBUG] 2023-05-23 13:30:23.394 [main] registry - Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x58649b7e @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[DEBUG] 2023-05-23 13:30:23.497 [LogStash::Runner] runner - -------- Logstash Settings (* means modified) ---------
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - node.name: "Yasser-Dell"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - *path.config: "/etc/logstash/conf.d/logstash.conf"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 []>
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_list: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_variable_list: []
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - modules_setup: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.test_and_exit: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.reload.automatic: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0xdeddccb>
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.support_escapes: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - config.field_reference.parser: "STRICT"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - metric.collect: true
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.id: "main"
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.system: false
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.workers: 20
[DEBUG] 2023-05-23 13:30:23.498 [LogStash::Runner] runner - pipeline.batch.size: 125
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.batch.delay: 50
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.unsafe_shutdown: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.java_execution: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.reloadable: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.plugin_classloaders: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.separate_logs: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.ordered: "auto"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - pipeline.ecs_compatibility: "disabled"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - path.plugins: []
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - *config.debug: true (default: false)
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - *log.level: "debug" (default: "info")
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - version: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - help: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - enable-local-plugin-development: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - log.format: "plain"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.enabled: true
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.http.host: "127.0.0.1"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.http.port: 9600..9700
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.environment: "production"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.auth.type: "none"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - api.ssl.enabled: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.type: "memory"
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.drain: false
[DEBUG] 2023-05-23 13:30:23.499 [LogStash::Runner] runner - queue.page_capacity: 67108864
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.max_bytes: 1073741824
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.max_events: 0
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.acks: 1024
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.writes: 1024
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.interval: 1000
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - queue.checkpoint.retry: true
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.enable: false
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.max_bytes: 1073741824
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - dead_letter_queue.flush_interval: 5000
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x11d16eb8>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x367fd31a>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x59c68ed3>
[DEBUG] 2023-05-23 13:30:23.500 [LogStash::Runner] runner - slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x7459002d>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - keystore.file: "/usr/share/logstash/config/logstash.keystore"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - path.settings: "/usr/share/logstash/config"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.enabled: false
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x157eac27>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x44cefd71>
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.501 [LogStash::Runner] runner - xpack.monitoring.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - xpack.monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - xpack.monitoring.collection.config.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.enabled: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x346c568d>
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x3c9b416a>
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.pipeline.details.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - monitoring.collection.config.enabled: true
[DEBUG] 2023-05-23 13:30:23.502 [LogStash::Runner] runner - node.uuid: ""
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.enabled: false
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x4de37f33>
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.pipeline.id: ["main"]
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.username: "logstash_system"
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - xpack.management.elasticsearch.sniffing: false
[DEBUG] 2023-05-23 13:30:23.503 [LogStash::Runner] runner - --------------- Logstash Settings -------------------
[WARN ] 2023-05-23 13:30:23.515 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[DEBUG] 2023-05-23 13:30:23.533 [LogStash::Runner] agent - Initializing API WebServer {"api.http.host"=>"127.0.0.1", "api.http.port"=>9600..9700, "api.ssl.enabled"=>false, "api.auth.type"=>"none", "api.environment"=>"production"}
[DEBUG] 2023-05-23 13:30:23.548 [LogStash::Runner] service - [api-service] start
[DEBUG] 2023-05-23 13:30:23.584 [LogStash::Runner] agent - Setting up metric collection
[DEBUG] 2023-05-23 13:30:23.607 [LogStash::Runner] os - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.725 [LogStash::Runner] jvm - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.766 [LogStash::Runner] jvm - collector name {:name=>"G1 Young Generation"}
[DEBUG] 2023-05-23 13:30:23.768 [LogStash::Runner] jvm - collector name {:name=>"G1 Old Generation"}
[DEBUG] 2023-05-23 13:30:23.776 [LogStash::Runner] persistentqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.782 [LogStash::Runner] deadletterqueue - Starting {:polling_interval=>5, :polling_timeout=>120}
[DEBUG] 2023-05-23 13:30:23.995 [Agent thread] agent - Starting agent
[DEBUG] 2023-05-23 13:30:24.001 [Api Webserver] agent - Starting API WebServer (puma)
[DEBUG] 2023-05-23 13:30:24.027 [Agent thread] configpathloader - Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf.d/02-beats-input.conf", "/etc/logstash/conf.d/30-elasticsearch-output.conf", "/etc/logstash/conf.d/grok-example.conf", "/etc/logstash/conf.d/logstash-simple.conf", "/etc/logstash/conf.d/logstash.yml", "/etc/logstash/conf.d/logstash_apache.conf", "/etc/logstash/conf.d/logstash_grok.conf"]}
[DEBUG] 2023-05-23 13:30:24.028 [Agent thread] configpathloader - Reading config file {:config_file=>"/etc/logstash/conf.d/logstash.conf"}
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - -------- Logstash Config ---------
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - Config from source, source: LogStash::Config::Source::Local, pipeline_id:: main
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig - Config string, protocol: file, id: /etc/logstash/conf.d/logstash.conf
[DEBUG] 2023-05-23 13:30:24.050 [Agent thread] PipelineConfig -

input
{
  file
  {
    path => "/home/yasser/balance_inquiry.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    sincedb_clean_after => "60 s"
  }
}

output
{
  elasticsearch
  {
    hosts => "localhost:9200"
    index => "balance-inquiry"
  }
  stdout
  {
    codec => rubydebug
  }
}

[DEBUG] 2023-05-23 13:30:24.051 [Agent thread] PipelineConfig - Merged config
[DEBUG] 2023-05-23 13:30:24.051 [Agent thread] PipelineConfig -

input
{
  file
  {
    path => "/home/yasser/balance_inquiry.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    sincedb_clean_after => "60 s"
  }
}

output
{
  elasticsearch
  {
    hosts => "localhost:9200"
    index => "balance-inquiry"
  }
  stdout
  {
    codec => rubydebug
  }
}

[DEBUG] 2023-05-23 13:30:24.055 [Api Webserver] agent - Trying to start API WebServer {:port=>9600, :ssl_enabled=>false}[INFO ] 2023-05-23 13:30:24.074 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[DEBUG] 2023-05-23 13:30:24.085 [Agent thread] agent - Converging pipelines state {:actions_count=>1}
[DEBUG] 2023-05-23 13:30:24.092 [Converge PipelineAction::Create<main>] agent - Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[DEBUG] 2023-05-23 13:30:24.101 [Converge PipelineAction::Create<main>] SecretStoreFactory - Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[DEBUG] 2023-05-23 13:30:24.384 [Converge PipelineAction::Create<main>] Reflections - going to scan these urls:
jar:file:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar!/
[INFO ] 2023-05-23 13:30:24.432 [Converge PipelineAction::Create<main>] Reflections - Reflections took 47 ms to scan 1 urls, producing 119 keys and 419 values
[DEBUG] 2023-05-23 13:30:24.437 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.JsonDeserializer -> com.fasterxml.jackson.databind.deser.std.StdDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.deser.NullValueProvider -> com.fasterxml.jackson.databind.JsonDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> com.fasterxml.jackson.databind.deser.std.StdDeserializer
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.AbstractFilterable$Builder -> org.apache.logging.log4j.core.appender.AbstractAppender$Builder
[DEBUG] 2023-05-23 13:30:24.438 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.security.SecureClassLoader -> java.net.URLClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.ClassLoader -> java.security.SecureClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Closeable -> java.net.URLClassLoader
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.AutoCloseable -> java.io.Closeable
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonFormatVisitors.JsonFormatVisitable -> com.fasterxml.jackson.databind.JsonSerializer
[DEBUG] 2023-05-23 13:30:24.439 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.config.builder.api.ConfigurationBuilderFactory -> org.apache.logging.log4j.core.config.ConfigurationFactory[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.RubyObject -> org.jruby.RubyException
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.RubyBasicObject -> org.jruby.RubyObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.IRubyObject -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Comparable -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.marshal.CoreObjectType -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.InstanceVariables -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.jruby.runtime.builtin.InternalVariables -> org.jruby.RubyBasicObject
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.ser.std.StdSerializer -> com.fasterxml.jackson.databind.ser.std.StdScalarSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.JsonSerializer -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonFormatVisitors.JsonFormatVisitable -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype com.fasterxml.jackson.databind.jsonschema.SchemaAware -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> com.fasterxml.jackson.databind.ser.std.StdSerializer
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Throwable -> java.lang.Exception
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.lang.Throwable
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractList -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractCollection -> java.util.AbstractList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Collection -> java.util.AbstractCollection
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Iterable -> java.util.Collection
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.List -> java.util.AbstractList
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Collection -> java.util.List
[DEBUG] 2023-05-23 13:30:24.440 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.List -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.RandomAccess -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.util.ArrayList
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.AbstractFilterable -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.AbstractLifeCycle -> org.apache.logging.log4j.core.filter.AbstractFilterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle2 -> org.apache.logging.log4j.core.AbstractLifeCycle
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.LifeCycle2
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.filter.Filterable -> org.apache.logging.log4j.core.filter.AbstractFilterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.filter.Filterable
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.Appender -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.Appender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.impl.LocationAware -> org.apache.logging.log4j.core.appender.AbstractAppender
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Exception -> java.io.IOException
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Exception -> java.lang.RuntimeException
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.message.MessageFactory -> org.apache.logging.log4j.message.MessageFactory2
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.AbstractLifeCycle -> org.apache.logging.log4j.core.filter.AbstractFilter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.Filter -> org.apache.logging.log4j.core.filter.AbstractFilter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LifeCycle -> org.apache.logging.log4j.core.Filter
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.constant.Constable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.441 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Comparable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.lang.Enum
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.LogEvent -> org.apache.logging.log4j.core.impl.Log4jLogEvent
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.apache.logging.log4j.core.LogEvent
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.AbstractMap -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Map -> java.util.AbstractMap
[DEBUG] 2023-05-23 13:30:24.442 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.util.Map -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.Cloneable -> java.util.IdentityHashMap
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype org.apache.logging.log4j.core.impl.LocationAwareLogEventFactory -> org.apache.logging.log4j.core.impl.LogEventFactory
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.io.Serializable -> org.apache.logging.log4j.message.Message
[DEBUG] 2023-05-23 13:30:24.443 [Converge PipelineAction::Create<main>] Reflections - expanded subtype java.lang.RuntimeException -> java.lang.IllegalArgumentException
[DEBUG] 2023-05-23 13:30:24.449 [Converge PipelineAction::Create<main>] SecretStoreFactory - Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[DEBUG] 2023-05-23 13:30:24.564 [Converge PipelineAction::Create<main>] registry - On demand adding plugin to the registry {:name=>"file", :type=>"input", :class=>LogStash::Inputs::File}
[DEBUG] 2023-05-23 13:30:24.589 [Converge PipelineAction::Create<main>] registry - On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
[DEBUG] 2023-05-23 13:30:24.645 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@id = "plain_fe49ab78-3618-43c6-83d9-f995b6fa59b8"
[DEBUG] 2023-05-23 13:30:24.645 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@enable_metric = true
[DEBUG] 2023-05-23 13:30:24.646 [Converge PipelineAction::Create<main>] plain - config LogStash::Codecs::Plain/@charset = "UTF-8"
[WARN ] 2023-05-23 13:30:24.650 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[DEBUG] 2023-05-23 13:30:24.674 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@start_position = "beginning"
[DEBUG] 2023-05-23 13:30:24.674 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@path = ["/home/yasser/balance_inquiry.log"]
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_clean_after = 60.0
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@id = "c22aed2b5b294f372c697137c06beb4a02f6fc8367040c748d70c7cfc569900d"
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_path = "/dev/null"
[DEBUG] 2023-05-23 13:30:24.675 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@enable_metric = true
[DEBUG] 2023-05-23 13:30:24.678 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@codec = <LogStash::Codecs::Plain id=>"plain_fe49ab78-3618-43c6-83d9-f995b6fa59b8", enable_metric=>true, charset=>"UTF-8">
[DEBUG] 2023-05-23 13:30:24.678 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@add_field = {}
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@stat_interval = 1.0
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@discover_interval = 15
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@sincedb_write_interval = 15.0
[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create<main>] file - config LogStash::Inputs::File/@delimiter = "\n"

type or paste code here

26

[DEBUG] 2023-05-23 13:30:24.679 [Converge PipelineAction::Create] file - config LogStash::Inputs::File/@delimiter = "\n"

Is this the last line? Should be more

27

sorry for many messages, but it seems issue is resolved!
just to conclude, what I did:

  • create a folder at /usr/share/logstash/config
  • move logstash.yml at this config folder
  • grant permission to folder /var/lib/logstash and all hierarchy

now the index is created and data is moved.

The question here:

  • what was the issue?
  • why this configration folder is not automatically created and why the file logstash.yml is automatically created at differetn folder /etc/logstash instead of expected /usr/share/logstash/config?!
  • what is the problem of permission, should I install logstash using sudo or without it?
yasser@Yasser-Dell:/usr/share/logstash/bin$ curl -XGET 127.0.0.1:9200/balance-inquiry/_search?pretty
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 62,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "balance-inquiry",
        "_id" : "EgNLSIgB_xR7YnnXbwgI",
        "_score" : 1.0,
        "_source" : {
          "message" : "  customerType=1",
          "host" : "Yasser-Dell",
          "@version" : "1",
          "path" : "/home/yasser/balance_inquiry.log",
          "@timestamp" : "2023-05-23T11:08:02.061Z"
        }
      },
      {
        "_index" : "balance-inquiry",
        "_id" : "FQNLSIgB_xR7YnnXbwgI",
        "_score" : 1.0,
        "_ignored" : [
          "message.keyword"
        ],
        "_source" : {
          "message" : "INFO: final balanceInquiryReplyVO = [BalanceInquiryReplyVO [header=MessageHeaderVO [msgFormat=null, msgVersion=null, msgOption=null, requestorChannelID=null, replierChannelID=null, requestorChannelFunction=null, requestorUserID=null, lang=null, securityInfo=null, securityInfoKey=null, returnCode=0000, errorCode=null, errorMessage=0000, srDate=null, srRcvDate=null, srStatus=0, funcId=null, securityKey=null, requestorLanguage=null, overwriteOpenOrder=null, chargeable=null, serviceRequestId=null, backendChannel=null], lineNumber=966530507255, customerType=1, packageID=Mobily Prepaid MNP, balance=0.75, expirationDate=Sat Aug 12 00:00:00 GMT+03:00 2023, billedAmount=0.0, unbilledAmount=0.0, unalocatedAmount=0.0, inValue=0, accountNumber=null, outstandingAmount=0.0, amountDue=0.0, packageName=null, monthlySubscriptionFee=0.0, lastBillDate=null, lastUpdateTime=null, additionalRentalFees=0.0, renewalAmount=0.0, ftthRenewexpDate=null, tempFtthRenewexpDate=null, errorCode=null, errorMessage=null, ncraBalance=null, unAllocated=0.0]]",
28

@Yasser_Alsawy

Good job trudging through this!

And now like I said I think it was something simple that we all did not see.

Did you install as a package?

And then you were running from the command line.

And like I said I should have seen this...

If you install as a package then run as a command line you have to add additional parameters to tell it where the config is at.

--path.settings /etc/logstash

This tells logstash where logstash.yml and pipeline.yml are at.

So the command should be

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf -r --path.settings /etc/logstash

If you look in the service startup file, you will see the command look like that.

If you had used tar.gz then your command would have worked.

I think if you set yourself all the way back and ran it like this it would work..

Also if you started it via the service it would work.

Now if you start logstash via the service /systemctl after you've moved everything you're going to run into another problem because they were looking /etc/logstash... And you will be out of sync and other frustrating situation may occur

What I would probably do is save off all your config files.... Completely uninstall the package and make sure it cleans up everything because now you moved files around.

Reinstall put all your correct config in the right place.

Try the command I showed above.

Then try starting as a service.

Apologies... Should have seen that

1 Like
29

trace != debug. filewatch provides helpful logging only at trace level.

1 Like
30

Thanks a lot @stephenb for clarifying the impact of installing as a package and running as a service. actually there is a limitation at WSL in tunning as a service (I run Elasticsearch as a service but neither kibana nor lostash). I have reinstalled all 3 products from tar file not debian but still have this shortage at kibana and logstash. I'll work on this any way.

only last question before closing this topic, I never edited pipelines.yml and I do not know where it should be located. is it impacted or created by logstash.conf or should it have different type of configuration?
I saw many videos talking about pipelines while they edit input and output at only logstash.conf. This is why it is confusing a little.

31

I suggest you review the directory layouts for logstash installations

In short, if you install as a tar.gz everything is located below the extracted directory.

If you install as a package, the configuration is in /etc/logstash and below and the binaries and logs etc. Where they should be.

And let's be really precise and clear again. I'm not as familiar with the Ubuntu on windows.

On Ubuntu you run as a service with systemctl , When I say service that is what I mean.

Not a window service.

Apologies, I don't know what that means...

So if you install by tar.gz should look like

hyperion:logstash-8.7.0 sbrown$ pwd
/Users/sbrown/workspace/elastic-install/8.7.0/logstash-8.7.0
hyperion:logstash-8.7.0 sbrown$ ls
CONTRIBUTORS                    NOTICE.TXT                      lib/                            tools/
Gemfile                         bin/                            logs/                           vendor/
Gemfile.lock                    config/                         logstash-core/                  x-pack/
JDK_VERSION                     data/                           logstash-core-plugin-api/
LICENSE.txt                     jdk.app/                        modules/
hyperion:logstash-8.7.0 sbrown$ ls config/
jvm.options             logstash-sample.conf    pipelines.yml
log4j2.properties       logstash.yml            startup.options
hyperion:logstash-8.7.0 sbrown$

if you install as a package

[root@stephenb-centos-logstash ~]# cd /etc/logstash
[root@stephenb-centos-logstash logstash]# ls -lrt
total 40
-rw-r--r--. 1 root root   285 Feb 12 05:39 pipelines.yml
-rw-r--r--. 1 root root   342 Feb 12 05:39 logstash-sample.conf
-rw-r--r--. 1 root root  7437 Feb 12 05:39 log4j2.properties
-rw-r--r--. 1 root root  1833 Feb 12 05:39 jvm.options
-rw-r--r--. 1 root root 15011 Mar 10 02:12 logstash.yml
-rw-------. 1 root root  1764 Mar 10 02:48 startup.options
drwxr-xr-x. 2 root root    34 Mar 10 02:52 conf.d
[root@stephenb-centos-logstash logstash]#

you use pipelines.yml see here if you want to run independent pipelines otherwise logstash will concatenate everything in the conf.d into 1 giant pipeline...

2 Likes
32

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.