# Logstash Start and shutdown intermittently **URL:** \ **Category:** Logstash\ **Tags:** elastic-stack-security\ **Created:** [October 13, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-start-and-shutdown-intermittently/251836 "2020-10-13T04:15:50Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Amit\_K\_Yadav](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@Amit\_K\_Yadav](https://discuss.elastic.co/u/Amit_K_Yadav)\ **Post date:** [October 13, 2020, 4:15am UTC](https://discuss.elastic.co/t/logstash-start-and-shutdown-intermittently/251836/1 "2020-10-13T04:15:51Z")
I am new to ELK world. I installed ELK, each on different server. My Elastic Stack was working fine before x-pack enabled. After enabling x-pack, Elasticsearch and kibana working fine. Logstash start and shutdown repeatedly.. Here is my Logstash.yml file ```auto # **path.data: /var/lib/logstash** **pipeline.ordered: auto** **log.level: debug #info** **path.logs: /var/log/logstash** **#** **xpack.monitoring.enabled: true** **xpack.monitoring.elasticsearch.username: logstash_system** **xpack.monitoring.elasticsearch.password: logstash_pwd** **#xpack.monitoring.elasticsearch.hosts: ["https://my-internal-DNS:9200"]** **#xpack.monitoring.elasticsearch.ssl.certificate_authority: "/etc/logstash/elastic-stack-ca-logstash.pem"** **#xpack.monitoring.elasticsearch.ssl.verification_mode: none** **#certificate** **#xpack.monitoring.elasticsearch.sniffing: false** **#** ``` Here is my pipeline file ```auto **input {** **beats {** **port => 5044** **}** **}** **output {** **elasticsearch {** **hosts => ["https://my-internal-DNS:9200"]** **index => "%{[@metadata][beat]}-%{[@metadata][version]}"** **user => "${ES_USER}"** **password => "${ES_PWD}"** **}** **}** ``` Here is log output. ```auto **[2020-10-13T04:16:59,785][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/demo.conf"}** **[2020-10-13T04:16:59,842][DEBUG][logstash.agent] Converging pipelines state {:actions_count=>1}** **[2020-10-13T04:16:59,859][DEBUG][logstash.agent] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}** **[2020-10-13T04:17:00,310][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"{\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 13, column 14 (byte 275) after output {\n elasticsearch {\n hosts => [\"http://10.0.4.4:9200\"]\n# hosts => [\"https://prod-elasticsearch.leadschool.infra:9200\"]\n index => \"%{[@metadata][beat]}-%{[@metadata][version]}\" \n user => #\"${ES_USER}\"\n password ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}** **[2020-10-13T04:17:00,373][DEBUG][logstash.agent] Starting puma** **[2020-10-13T04:17:00,390][DEBUG][logstash.instrument.periodicpoller.os] Stopping** **[2020-10-13T04:17:00,405][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping** **[2020-10-13T04:17:00,406][DEBUG][logstash.agent] Trying to start WebServer {:port=>9600}** **[2020-10-13T04:17:00,409][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping** **[2020-10-13T04:17:00,419][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping** **[2020-10-13T04:17:00,431][DEBUG][logstash.agent] Shutting down all pipelines {:pipelines_count=>0}** **[2020-10-13T04:17:00,459][DEBUG][logstash.agent] Converging pipelines state {:actions_count=>0}** **[2020-10-13T04:17:00,461][DEBUG][logstash.api.service] [api-service] start** **[2020-10-13T04:17:00,622][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}** **[2020-10-13T04:17:05,513][INFO][logstash.runner] Logstash shut down.** **[2020-10-13T04:17:05,540][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit** **[2020-10-13T04:17:05,548][DEBUG][logstash.agent] Error in reactor loop escaped: closed stream (IOError)** ``` But my Elasticsearch is reachable from curl.. ```auto **root@logstash-01:/etc/logstash# curl -k https://my-internal-DNS:9200/_cluster/health?pretty -u 'elastic'** **Enter host password for user 'elastic':** **{** **"cluster_name" : "monitor",** **"status" : "green",** **"timed_out" : false,** **"number_of_nodes" : 3,** **"number_of_data_nodes" : 3,** **"active_primary_shards" : 7,** **"active_shards" : 14,** **"relocating_shards" : 0,** **"initializing_shards" : 0,** **"unassigned_shards" : 0,** **"delayed_unassigned_shards" : 0,** **"number_of_pending_tasks" : 0,** **"number_of_in_flight_fetch" : 0,** **"task_max_waiting_in_queue_millis" : 0,** **"active_shards_percent_as_number" : 100.0** **}** **root@logstash-01:/etc/logstash#** ``` Any help in this will be great. Thanks. ---
**Author:** ![Amit\_K\_Yadav](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@Amit\_K\_Yadav](https://discuss.elastic.co/u/Amit_K_Yadav)\ **Post date:** [October 13, 2020, 4:29am UTC](https://discuss.elastic.co/t/logstash-start-and-shutdown-intermittently/251836/2 "2020-10-13T04:29:30Z")
If I pass username password to pipeline ```auto **user => "elastic"** **password => "elasticpassword"** ``` instated of this ```auto **user => "${ES_USER}"** **password => "${ES_PWD}"** ``` I am getting this error.... ```auto **[2020-10-13T04:23:48,750][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}** **[2020-10-13T04:23:48,871][DEBUG][logstash.outputs.elasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 4s** **[2020-10-13T04:23:51,836][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/, :path=>"/"}** **[2020-10-13T04:23:51,873][WARN][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}** **[2020-10-13T04:23:52,689][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}** **[2020-10-13T04:23:52,695][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}** **[2020-10-13T04:23:52,873][DEBUG][logstash.outputs.elasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 8s** **[2020-10-13T04:23:56,877][DEBUG][logstash.outputs.elasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://elastic:xxxxxx@prod-elasticsearch.leadschool.infra:9200/, :path=>"/"}** ``` Now My elasticsearch is reachable from curl but not to Logstash ..... Confused as F\*\*k..... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 10, 2020, 4:29am UTC](https://discuss.elastic.co/t/logstash-start-and-shutdown-intermittently/251836/3 "2020-11-10T04:29:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.