Unable to monitor logstash from kibana

Hi Team,

I am trying to monitor logstash from Kibana. Ihave modified the logstash.yml file but its not adding to kibana / not able to see it from kibana.

My Elasticsearch is running with TLS.

added below parameters in logstash.yml file.

# X-Pack Monitoring
# https://www.elastic.co/guide/en/logstash/current/monitoring-logstash.html
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: XXXXXXXX
#xpack.monitoring.elasticsearch.proxy: ["https://XX.XX.XX.XX:9200"]
xpack.monitoring.elasticsearch.hosts: ["https://XX.XX.XX.XX:9200", "https://XX.XX.XX.XX:9200","https://XX.XX.XX.XX:9200"]
# an alternative to hosts + username/password settings is to use cloud_id/cloud_auth
#xpack.monitoring.elasticsearch.cloud_id: monitoring_cluster_id:xxxxxxxxxx
#xpack.monitoring.elasticsearch.cloud_auth: logstash_system:password
# another authentication alternative is to use an Elasticsearch API key
#xpack.monitoring.elasticsearch.api_key: "id:api_key"
xpack.monitoring.elasticsearch.ssl.certificate_authority: [ "/etc/logstash/elk-stack.crt.pem" ]
xpack.monitoring.elasticsearch.ssl.truststore.path: /etc/logsatsh/cert.jks
xpack.monitoring.elasticsearch.ssl.truststore.password: password
#xpack.monitoring.elasticsearch.ssl.keystore.path: /path/to/file

i am getting below error if i run the systemctl start logstash command

systemctl status is showing running ..

[root@lnitiblbk2u logstash]# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/usr/lib/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2022-07-15 13:32:36 +04; 447ms ago
 Main PID: 24990 (java)
   CGroup: /system.slice/logstash.service
           └─24990 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedy...

Jul 15 13:32:36 lnitiblbk2u systemd[1]: Started logstash.
Jul 15 13:32:36 lnitiblbk2u logstash[24990]: Using bundled JDK: /usr/share/logstash/jdk
Jul 15 13:32:36 lnitiblbk2u logstash[24990]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

logstash logs -

[2022-07-15T13:33:13,338][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-07-15T13:33:13,401][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.3.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"}
[2022-07-15T13:33:13,403][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-07-15T13:33:13,417][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:91) ~[?:?]

but its not added to kibana.

You have a FATAL error when starting Logstash, your Logstash is not running. Do you have anything else in the logs after that? Check /var/log/messages or /var/log/syslog and search for logstash to see if you have some hint of why logstash didn't start.

Also, share your logstash.yml file.