Port configured for logstash does not turn up (5044)

Manula_Manjitha · October 2, 2023, 9:09am

I have configured a logstash on my server and the configurations in /etc/logstash/conf.d/beats.conf file are as follows.

input {
  beats {
    port => 5044
    host => "192.168.14.189"
  }
}

filter {
  if [type] =="syslog" {
     grok {
        match => { "message" => "%{SYSLOGLINE}" }
  }
     date {
        match => [ "timestamp", "MMM  d HH:mm:ss" ]
     }
  }
}

output {
  elasticsearch {
    hosts => ["192.168.14.189:9200"]
    index => "%{[@metadata] [beat]}-%{+YYYY.MM.DD}"
  }
}

But when I start the service 5044 port is not turned up but when I hit systemctl status logstash.service , it shows me that service is running. When I tail the server journal while starting the logstash, it gives me the following error message.

Oct 02 14:28:01 ANBC-NTP-HA-SVR01 logstash[2907531]: [2023-10-02T14:28:01,154][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}

Can any one tell me the reason for this?

Rios · October 2, 2023, 10:12am

Welcome to the community.

The beats.conf file looks OK, I have tested on my side. Since you are running the service mode, do you have another .conf file(s) in the same directory?
The service read and run all .conf files from /etc/logstash/conf.d/

Manula_Manjitha · October 2, 2023, 10:17am

@Rios Thanks for the reply. No, currently this is the only one running at the place.

Rios · October 2, 2023, 10:19am

Can you change: host => "0.0.0.0" and test?

Manula_Manjitha · October 2, 2023, 10:39am

@Rios Same error comes..

Rios · October 2, 2023, 11:34am

Remove space between
%{[@metadata][beat]}

Manula_Manjitha · October 2, 2023, 11:37am

@Rios Still it's same..And the error says something to be done at line 5..I can't identify what is that..

leandrojmp · October 2, 2023, 11:51am

As already mentioned, there is no error in your configuration file, so some thing is not right with your file or your service.

Restart the service to get new logs and share all the logstash log lines from when Logstash start and until it fails to start.

Did you edit it directly on your server or on an external editor and pasted the content?

Since beats.conf is the only file inside /etc/logstash/conf.d and it is complaining about an error on line 5, which is in your input, I would remove your input and type everything again.

Manula_Manjitha · October 2, 2023, 11:55am

@leandrojmp Already done that too..But nothing seems to be working..

leandrojmp · October 2, 2023, 11:56am

Please share the Logstash logs as asked.

The logs from when logstash start can give more insight in your issue, but you need to try to start the service again to get fresh logs.

Manula_Manjitha · October 2, 2023, 12:14pm

@leandrojmp

[2023-10-02T17:34:08,120][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-10-02T17:34:08,131][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2023-10-02T17:34:08,133][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-10-02T17:34:08,138][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2023-10-02T17:34:08,139][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x47cdae47 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2023-10-02T17:34:08,140][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2023-10-02T17:34:08,140][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x4d026a3a @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2023-10-02T17:34:08,371][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2023-10-02T17:34:08,373][DEBUG][logstash.runner          ] node.name: "ANBC-NTP-HA-SVR01"
[2023-10-02T17:34:08,374][DEBUG][logstash.runner          ] *path.config: "/etc/logstash/pipelines.yml"
[2023-10-02T17:34:08,374][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2023-10-02T17:34:08,374][DEBUG][logstash.runner          ] modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 []>
[2023-10-02T17:34:08,374][DEBUG][logstash.runner          ] modules: []
[2023-10-02T17:34:08,374][DEBUG][logstash.runner          ] modules_list: []
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] modules_variable_list: []
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] modules_setup: false
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] config.test_and_exit: false
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] config.reload.automatic: false
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x5dea8bf2>
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] config.support_escapes: false
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] metric.collect: true
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] pipeline.id: "main"
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] pipeline.system: false
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] *pipeline.workers: 16 (default: 4)
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] *pipeline.batch.size: 2048 (default: 125)
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] *pipeline.batch.delay: 100 (default: 50)
[2023-10-02T17:34:08,375][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] pipeline.reloadable: true
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] pipeline.separate_logs: false
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] pipeline.ordered: "auto"
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] pipeline.ecs_compatibility: "v8"
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] path.plugins: []
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] config.debug: false
[2023-10-02T17:34:08,376][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] version: false
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] help: false
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] enable-local-plugin-development: false
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] log.format: "plain"
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] api.enabled: true
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] api.http.host: "127.0.0.1"
[2023-10-02T17:34:08,377][DEBUG][logstash.runner          ] api.http.port: 9600..9700
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] api.environment: "production"
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] api.auth.type: "none"
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] api.ssl.enabled: false
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.type: "memory"
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.drain: false
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.max_events: 0
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] queue.checkpoint.retry: true
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
[2023-10-02T17:34:08,378][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] dead_letter_queue.flush_interval: 5000
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x539f0729>
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x3fa27158>
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x6026c754>
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x11f8bff1>
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x99caac2>
[2023-10-02T17:34:08,379][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x776609b>
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] monitoring.enabled: false
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x154bc3cf>
[2023-10-02T17:34:08,380][DEBUG][logstash.runner          ] monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x29862f60>
[2023-10-02T17:34:08,381][DEBUG][logstash.runner          ] monitoring.elasticsearch.username: "logstash_system"
[2023-10-02T17:34:08,381][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.verification_mode: "certificate"
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] monitoring.elasticsearch.sniffing: false
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] monitoring.collection.pipeline.details.enabled: true
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] monitoring.collection.config.enabled: true
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] node.uuid: ""
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x35b1e5e5>
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2023-10-02T17:34:08,382][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2023-10-02T17:34:08,410][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-10-02T17:34:08,446][DEBUG][logstash.agent           ] Initializing API WebServer {"api.http.host"=>"127.0.0.1", "api.http.port"=>9600..9700, "api.ssl.enabled"=>false, "api.auth.type"=>"none", "api.environment"=>"production"}
[2023-10-02T17:34:08,476][DEBUG][logstash.api.service     ] [api-service] start
[2023-10-02T17:34:08,557][DEBUG][logstash.agent           ] Setting up metric collection
[2023-10-02T17:34:08,603][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling_interval=>5, :polling_timeout=>120}
[2023-10-02T17:34:08,789][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling_interval=>5, :polling_timeout=>120}
[2023-10-02T17:34:08,863][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2023-10-02T17:34:08,868][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2023-10-02T17:34:08,884][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
[2023-10-02T17:34:08,895][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
[2023-10-02T17:34:09,272][DEBUG][logstash.agent           ] Starting agent
[2023-10-02T17:34:09,296][DEBUG][logstash.agent           ] Starting API WebServer (puma)
[2023-10-02T17:34:09,349][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf.d", "/etc/logstash/jvm.options", "/etc/logstash/log4j2.properties", "/etc/logstash/logstash-sample", "/etc/logstash/logstash.yml", "/etc/logstash/startup.options"]}
[2023-10-02T17:34:09,356][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/pipelines.yml"}
[2023-10-02T17:34:09,378][DEBUG][logstash.agent           ] Trying to start API WebServer {:port=>9600, :ssl_enabled=>false}
[2023-10-02T17:34:09,409][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-10-02T17:34:09,448][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2023-10-02T17:34:09,464][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[2023-10-02T17:34:09,491][DEBUG][org.logstash.secret.store.SecretStoreFactory] Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
[2023-10-02T17:34:09,743][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
[2023-10-02T17:34:09,788][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2023-10-02T17:34:09,792][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2023-10-02T17:34:09,796][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2023-10-02T17:34:09,804][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2023-10-02T17:34:09,805][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2023-10-02T17:34:09,805][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2023-10-02T17:34:09,849][DEBUG][logstash.agent           ] API WebServer has stopped running
[2023-10-02T17:34:09,851][INFO ][logstash.runner          ] Logstash shut down.
[2023-10-02T17:34:09,860][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

Manula_Manjitha · October 2, 2023, 12:17pm

@leandrojmp

-- Logs begin at Mon 2023-10-02 17:27:44 +0530. --
Oct 02 17:28:10 ANBC-NTP-HA-SVR01 systemd[1]: Started logstash.
Oct 02 17:28:10 ANBC-NTP-HA-SVR01 logstash[2980903]: Using bundled JDK: /usr/share/logstash/jdk
Oct 02 17:28:10 ANBC-NTP-HA-SVR01 logstash[2980903]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,453][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,464][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.2.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,467][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,472][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,474][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x47cdae47 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,476][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,476][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x4d026a3a @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,829][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] node.name: "ANBC-NTP-HA-SVR01"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] *path.config: "/etc/logstash/pipelines.yml"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] modules.cli: <Java::OrgLogstashUtil::ModulesSettingArray:1 []>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] modules: []
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] modules_list: []
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] modules_variable_list: []
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,830][DEBUG][logstash.runner          ] modules_setup: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] config.test_and_exit: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] config.reload.automatic: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] config.reload.interval: #<Java::OrgLogstashUtil::TimeValue:0x5dea8bf2>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] config.support_escapes: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] metric.collect: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] pipeline.id: "main"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] pipeline.system: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] *pipeline.workers: 16 (default: 4)
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] *pipeline.batch.size: 2048 (default: 125)
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] *pipeline.batch.delay: 100 (default: 50)
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,831][DEBUG][logstash.runner          ] pipeline.reloadable: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] pipeline.separate_logs: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] pipeline.ordered: "auto"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] pipeline.ecs_compatibility: "v8"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] path.plugins: []
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] config.debug: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] version: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] help: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] enable-local-plugin-development: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] log.format: "plain"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] api.enabled: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,835][DEBUG][logstash.runner          ] api.http.host: "127.0.0.1"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] api.http.port: 9600..9700
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] api.environment: "production"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] api.auth.type: "none"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] api.ssl.enabled: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.type: "memory"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.drain: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.max_events: 0
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,836][DEBUG][logstash.runner          ] queue.checkpoint.retry: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] dead_letter_queue.flush_interval: 5000
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] slowlog.threshold.warn: #<Java::OrgLogstashUtil::TimeValue:0x539f0729>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] slowlog.threshold.info: #<Java::OrgLogstashUtil::TimeValue:0x3fa27158>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] slowlog.threshold.debug: #<Java::OrgLogstashUtil::TimeValue:0x6026c754>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] slowlog.threshold.trace: #<Java::OrgLogstashUtil::TimeValue:0x11f8bff1>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,837][DEBUG][logstash.runner          ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x99caac2>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x776609b>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] monitoring.enabled: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] monitoring.elasticsearch.hosts: ["http://localhost:9200"]
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] monitoring.collection.interval: #<Java::OrgLogstashUtil::TimeValue:0x154bc3cf>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,838][DEBUG][logstash.runner          ] monitoring.collection.timeout_interval: #<Java::OrgLogstashUtil::TimeValue:0x29862f60>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] monitoring.elasticsearch.username: "logstash_system"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] monitoring.elasticsearch.ssl.verification_mode: "certificate"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] monitoring.elasticsearch.sniffing: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] monitoring.collection.pipeline.details.enabled: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] monitoring.collection.config.enabled: true
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] node.uuid: ""
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.enabled: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: #<Java::OrgLogstashUtil::TimeValue:0x35b1e5e5>
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "certificate"
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,839][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,870][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,905][DEBUG][logstash.agent           ] Initializing API WebServer {"api.http.host"=>"127.0.0.1", "api.http.port"=>9600..9700, "api.ssl.enabled"=>false, "api.auth.type"=>"none", "api.environment"=>"production"}
Oct 02 17:28:24 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:24,935][DEBUG][logstash.api.service     ] [api-service] start
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,021][DEBUG][logstash.agent           ] Setting up metric collection
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,072][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling_interval=>5, :polling_timeout=>120}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,289][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling_interval=>5, :polling_timeout=>120}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,360][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,364][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,383][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,395][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,770][DEBUG][logstash.agent           ] Starting agent
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,785][DEBUG][logstash.agent           ] Starting API WebServer (puma)
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,842][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/etc/logstash/conf.d", "/etc/logstash/jvm.options", "/etc/logstash/log4j2.properties", "/etc/logstash/logstash-sample", "/etc/logstash/logstash.yml", "/etc/logstash/startup.options"]}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,844][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/pipelines.yml"}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,891][DEBUG][logstash.agent           ] Trying to start API WebServer {:port=>9600, :ssl_enabled=>false}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,917][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,919][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,923][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
Oct 02 17:28:25 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:25,941][DEBUG][org.logstash.secret.store.SecretStoreFactory] Attempting to exists or secret store with implementation: org.logstash.secret.store.backend.JavaKeyStore
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,089][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 5, column 1 (byte 221) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:381:in `block in converge_state'"]}
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,137][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,139][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,143][DEBUG][logstash.instrument.periodicpoller.os] Stopping
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,150][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,151][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,152][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,192][DEBUG][logstash.agent           ] API WebServer has stopped running
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,193][INFO ][logstash.runner          ] Logstash shut down.
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: [2023-10-02T17:28:26,201][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 logstash[2980903]:         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 systemd[1]: logstash.service: Failed with result 'exit-code'.
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 systemd[1]: logstash.service: Service RestartSec=100ms expired, scheduling restart.
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Oct 02 17:28:26 ANBC-NTP-HA-SVR01 systemd[1]: Stopped logstash.

leandrojmp · October 2, 2023, 12:27pm

You are not running the config you think you are.

This is wrong, remove this line from your logstash.yml, this will tell logstash to run pipelines.yml as it was a logstash configuration file, which is not correct.

You should not have path.config in logstash.yml.

Just removing this line should solve the issue, but share both your logstash.yml and pipelines.yml to confirm this is the cause.

Manula_Manjitha · October 3, 2023, 8:11am

@leandrojmp You were right. When I point that path to my actual configurations, it worked.

system · October 31, 2023, 8:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.