Logstash Start and shutdown intermittently

[Amit_K_Yadav (Amit Kumar Yadav)]

I am new to ELK world. I installed ELK, each on different server. My Elastic Stack was working fine before x-pack enabled. After enabling x-pack, Elasticsearch and kibana working fine. Logstash start and shutdown repeatedly..

Here is my Logstash.yml file

#
**path.data: /var/lib/logstash**
**pipeline.ordered: auto**
**log.level: debug #info**
**path.logs: /var/log/logstash**
**#**
**xpack.monitoring.enabled: true**
**xpack.monitoring.elasticsearch.username: logstash_system**
**xpack.monitoring.elasticsearch.password: logstash_pwd**
**#xpack.monitoring.elasticsearch.hosts: ["https://my-internal-DNS:9200"]**
**#xpack.monitoring.elasticsearch.ssl.certificate_authority:  "/etc/logstash/elastic-stack-ca-logstash.pem" **
**#xpack.monitoring.elasticsearch.ssl.verification_mode: none**
**#certificate**
**#xpack.monitoring.elasticsearch.sniffing: false**
**#**

Here is my pipeline file

**input {**
**  beats {**
**    port => 5044**
**  }**
**}**

**output {**
**  elasticsearch {**
**    hosts => ["https://my-internal-DNS:9200"]**
**    index => "%{[@metadata][beat]}-%{[@metadata][version]}"**
**    user => "${ES_USER}"**
**    password => "${ES_PWD}"**
**  }**
**}**

Here is log output.

**[2020-10-13T04:16:59,785][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/demo.conf"}**
**[2020-10-13T04:16:59,842][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}**
**[2020-10-13T04:16:59,859][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}**
**[2020-10-13T04:17:00,310][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 13, column 14 (byte 275) after output {\n  elasticsearch {\n    hosts => [\"http://10.0.4.4:9200\"]\n#    hosts => [\"https://prod-elasticsearch.leadschool.infra:9200\"]\n    index => \"%{[@metadata][beat]}-%{[@metadata][version]}\" \n    user => #\"${ES_USER}\"\n    password ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}**
**[2020-10-13T04:17:00,373][DEBUG][logstash.agent           ] Starting puma**
**[2020-10-13T04:17:00,390][DEBUG][logstash.instrument.periodicpoller.os] Stopping**
**[2020-10-13T04:17:00,405][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping**
**[2020-10-13T04:17:00,406][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}**
**[2020-10-13T04:17:00,409][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping**
**[2020-10-13T04:17:00,419][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping**
**[2020-10-13T04:17:00,431][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}**
**[2020-10-13T04:17:00,459][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}**
**[2020-10-13T04:17:00,461][DEBUG][logstash.api.service     ] [api-service] start**
**[2020-10-13T04:17:00,622][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}**
**[2020-10-13T04:17:05,513][INFO ][logstash.runner          ] Logstash shut down.**
**[2020-10-13T04:17:05,540][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit**
**[2020-10-13T04:17:05,548][DEBUG][logstash.agent           ] Error in reactor loop escaped: closed stream (IOError)**

But my Elasticsearch is reachable from curl..

**root@logstash-01:/etc/logstash# curl -k  https://my-internal-DNS:9200/_cluster/health?pretty -u 'elastic'**
**Enter host password for user 'elastic':**
**{**
**  "cluster_name" : "monitor",**
**  "status" : "green",**
**  "timed_out" : false,**
**  "number_of_nodes" : 3,**
**  "number_of_data_nodes" : 3,**
**  "active_primary_shards" : 7,**
**  "active_shards" : 14,**
**  "relocating_shards" : 0,**
**  "initializing_shards" : 0,**
**  "unassigned_shards" : 0,**
**  "delayed_unassigned_shards" : 0,**
**  "number_of_pending_tasks" : 0,**
**  "number_of_in_flight_fetch" : 0,**
**  "task_max_waiting_in_queue_millis" : 0,**
**  "active_shards_percent_as_number" : 100.0**
**}**
**root@logstash-01:/etc/logstash#** 

Any help in this will be great. Thanks.

[Amit_K_Yadav (Amit Kumar Yadav)]

If I pass username password to pipeline

**user => "elastic"**
**    password => "elasticpassword"**

instated of this

**    user => "${ES_USER}"**
**    password => "${ES_PWD}"**

I am getting this error....

**[2020-10-13T04:23:48,750][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}**
**[2020-10-13T04:23:48,871][DEBUG][logstash.outputs.elasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 4s**
**[2020-10-13T04:23:51,836][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/, :path=>"/"}**
**[2020-10-13T04:23:51,873][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}**
**[2020-10-13T04:23:52,689][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}**
**[2020-10-13T04:23:52,695][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}**
**[2020-10-13T04:23:52,873][DEBUG][logstash.outputs.elasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 8s**
**[2020-10-13T04:23:56,877][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/, :path=>"/"}**

Now My elasticsearch is reachable from curl but not to Logstash .....
Confused as F**k.....