# Any solutions please! **URL:** \ **Category:** Logstash\ **Created:** [November 4, 2019, 7:40am UTC](https://discuss.elastic.co/t/any-solutions-please/206343 "2019-11-04T07:40:25Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 4, 2019, 7:40am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/1 "2019-11-04T07:40:25Z")
I have followed that link [https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logstash-filebeat-with-offline-install-in-linux](https://www.elastic.co/blog/how-to-setup-tls-for-elasticsearch-kibana-logstash-filebeat-with-offline-install-in-linux) As the result, I got the below error ``` [root@elastic logstash]# systemctl start logstash [root@elastic logstash]# systemctl status logstash ● logstash.service - logstash Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled) Active: active (running) since Mon 2019-11-04 02:37:04 EST; 49s ago Main PID: 23874 (java) CGroup: /system.slice/logstash.service └─23874 /bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupa... Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: logstash.service holdoff time over, scheduling restart. Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: Stopped logstash. Nov 04 02:37:04 elastic.nbc.org.kh.local systemd[1]: Started logstash. Nov 04 02:37:04 elastic.nbc.org.kh.local logstash[23874]: Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC ...ease. Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: An illegal reflective access operation has occurred Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Illegal reflective access by com.headius.backport9.modules.M...or.fd Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Please consider reporting this to the maintainers of com.hea...dules Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: Use --illegal-access=warn to enable warnings of further ille...tions Nov 04 02:37:28 elastic.nbc.org.kh.local logstash[23874]: WARNING: All illegal access operations will be denied in a future release Hint: Some lines were ellipsized, use -l to show in full. [root@elastic logstash]# less /var/log/logstash/logstash-plain.log [2019-11-04T01:23:42,909][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"} [2019-11-04T01:23:42,960][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"} [2019-11-04T01:23:43,814][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.2"} [2019-11-04T01:23:43,851][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"a6b62bd5-ca53-4339-936a-ef78e0065f7a", :path=>"/var/lib/logstash/uuid"} [2019-11-04T01:23:44,961][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} [2019-11-04T01:23:44,983][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2019-11-04T01:23:45,506][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-11-04T01:23:50,232][INFO][logstash.runner] Logstash shut down. [2019-11-04T01:25:45,886][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.2"} [2019-11-04T01:25:47,109][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} [2019-11-04T01:25:47,118][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2019-11-04T01:25:47,609][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-11-04T01:25:52,553][INFO][logstash.runner] Logstash shut down. [2019-11-04T01:26:43,565][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.2"} [2019-11-04T01:26:44,831][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} [2019-11-04T01:26:44,840][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2019-11-04T01:26:45,352][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-11-04T01:26:50,320][INFO][logstash.runner] Logstash shut down. [2019-11-04T01:27:26,683][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.2"} [2019-11-04T01:27:27,906][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} [2019-11-04T01:27:27,936][ERROR][logstash.config.sourceloader] No configuration found in the configured sources. [2019-11-04T01:27:28,419][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2019-11-04T01:27:33,406][INFO][logstash.runner] Logstash shut down. [2019-11-04T01:28:08,659][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.4.2"} [2019-11-04T01:28:09,772][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 4, 2019, 12:43pm UTC](https://discuss.elastic.co/t/any-solutions-please/206343/2 "2019-11-04T12:43:03Z")
> [@pbona](#): > > No config files found in path {:path=\>"/etc/logstash/conf.d/\*.conf"} What is unclear about that message? ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 5, 2019, 12:34am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/3 "2019-11-05T00:34:18Z")
> [@pbona](#): > > [ERROR][logstash.config.sourceloader] No configuration found in the configured sources. And not see the Pipeline started succesfully ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 5, 2019, 12:36am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/4 "2019-11-05T00:36:04Z")
> [@Badger](#): > > No config files found in path {:path=\>"/etc/logstash/conf.d/\*.conf"} The file is logstash.conf ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 5, 2019, 2:07am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/5 "2019-11-05T02:07:56Z")
``` [root@elastic logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2019-11-04 20:59:07.850 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2019-11-04 20:59:07.859 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.2"} [INFO] 2019-11-04 20:59:11.032 [Converge PipelineAction::Create
] Reflections - Reflections took 75 ms to scan 1 urls, producing 20 keys and 40 values [INFO] 2019-11-04 20:59:12.804 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}} [WARN] 2019-11-04 20:59:13.801 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"} [INFO] 2019-11-04 20:59:14.148 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN] 2019-11-04 20:59:14.151 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO] 2019-11-04 20:59:14.205 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]} [WARN] 2019-11-04 20:59:14.426 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [INFO] 2019-11-04 20:59:14.570 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#"} [INFO] 2019-11-04 20:59:14.944 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [INFO] 2019-11-04 20:59:15.167 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} [INFO] 2019-11-04 20:59:15.639 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"} warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true): LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash' perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80 perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278 with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277 Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285 exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341 rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359 maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91 setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10 setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52 ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 5, 2019, 2:08am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/6 "2019-11-05T02:08:03Z")
``` [FATAL] 2019-11-04 20:59:15.821 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]} [ERROR] 2019-11-04 20:59:16.166 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [INFO] 2019-11-04 20:59:16.303 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2019, 3:10pm UTC](https://discuss.elastic.co/t/any-solutions-please/206343/7 "2019-11-05T15:10:04Z")
> [@pbona](#): > > LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL '[https://172.16.5.199:9200/logstash](https://172.16.5.199:9200/logstash)' 403 is a Forbidden status. If you followed that blog post you generated passwords for various user accounts. You should have one of those configured on your elasticsearch output. It is not configured correctly. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 6, 2019, 7:31am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/8 "2019-11-06T07:31:23Z")
> [@Badger](#): > > 403 is a Forbidden status. If you followed that blog post you generated passwords for various user accounts. You should have one of those configured on your elasticsearch output. It is not configured correctly. Yes, but which user credential I should be put over there ? I have putted the correct _logstash\_system_ account and password ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 6, 2019, 8:34am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/9 "2019-11-06T08:34:46Z")
``` [FATAL] 2019-11-06 03:33:31.078 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]} [ERROR] 2019-11-06 03:33:31.253 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 6, 2019, 8:50am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/10 "2019-11-06T08:50:00Z")
``` [root@elastic ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2019-11-06 03:48:24.903 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2019-11-06 03:48:24.930 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.1"} [INFO] 2019-11-06 03:48:28.108 [Converge PipelineAction::Create
] Reflections - Reflections took 76 ms to scan 1 urls, producing 20 keys and 40 values [INFO] 2019-11-06 03:48:30.055 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}} [WARN] 2019-11-06 03:48:30.876 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"} [INFO] 2019-11-06 03:48:31.448 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN] 2019-11-06 03:48:31.451 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO] 2019-11-06 03:48:31.485 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]} [INFO] 2019-11-06 03:48:31.590 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [WARN] 2019-11-06 03:48:31.615 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [INFO] 2019-11-06 03:48:31.645 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#"} ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 6, 2019, 8:50am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/11 "2019-11-06T08:50:07Z")
``` [INFO] 2019-11-06 03:48:31.765 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true): LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash' perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80 perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278 with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277 Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285 exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341 rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359 maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91 setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10 setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52 [FATAL] 2019-11-06 03:48:32.017 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]} [ERROR] 2019-11-06 03:48:32.193 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit ``` ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 6:35am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/12 "2019-11-07T06:35:22Z")
@Badger Did you have any solutions? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2019, 6:45am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/13 "2019-11-07T06:45:47Z")
Plaese show your `logstash.yml` file as well as your pipeline configuration. ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 6:48am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/14 "2019-11-07T06:48:00Z")
``` [root@elastic ~]# cat /etc/logstash/logstash.yml | grep -v "^#" node.name: logstash.local path.data: /var/lib/logstash path.config: /etc/logstash/conf.d/*.conf path.logs: /var/log/logstash xpack.monitoring.enabled: true xpack.monitoring.elasticsearch.username: logstash_system xpack.monitoring.elasticsearch.password: 'WxUSo53xekkCaU2YcvVw' xpack.monitoring.elasticsearch.hosts: ['https://172.16.5.199:9200'] xpack.monitoring.elasticsearch.ssl.certificate_authority: /etc/logstash/config/certs/ca.crt [root@elastic ~]# cat /etc/logstash/pipelines.yml | grep -v "^#" - pipeline.id: main path.config: "/etc/logstash/conf.d/*.conf" [root@elastic ~]# ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2019, 6:49am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/15 "2019-11-07T06:49:29Z")
What does your logstash configuration look like? You should find this in `/etc/logstash/conf.d/`. Is it exactly what is in the blog post? Do you get any different error message if you disable monitoring by setting `xpack.monitoring.enabled` to `false`? ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 6:56am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/16 "2019-11-07T06:56:29Z")
``` [root@elastic ~]# cat /etc/logstash/conf.d/logstash.conf input { beats { port => 5044 ssl => true ssl_key => '/etc/logstash/config/certs/logstash.pkcs8.key' ssl_certificate => '/etc/logstash/config/certs/logstash.crt' } } output { elasticsearch { hosts => ["https://172.16.5.199:9200"] cacert => '/etc/logstash/config/certs/ca.crt' user => 'logstash_writer' password => 'WxUSo53xekkCaU2YcvVw' index => 'logstash-%{+YYYY.MM.dd}' } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 7, 2019, 6:58am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/17 "2019-11-07T06:58:28Z")
Does the `logstash_writer` user you created have exactly the same password as the `logstash_system` user? ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 7:03am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/18 "2019-11-07T07:03:54Z")
Yes, it is the same ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 7:05am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/19 "2019-11-07T07:05:51Z")
> [@Christian\_Dahlqvist](#): > > Do you get any different error message if you disable monitoring by setting `xpack.monitoring.enabled` to `false` ? Let me test first ---
**Author:** ![pbona](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pbona/32/54832_2.png) [@pbona](https://discuss.elastic.co/u/pbona)\ **Post date:** [November 7, 2019, 7:09am UTC](https://discuss.elastic.co/t/any-solutions-please/206343/20 "2019-11-07T07:09:35Z")
> [@Christian\_Dahlqvist](#): > > Do you get any different error message if you disable monitoring by setting `xpack.monitoring.enabled` to `false` ? ``` [root@elastic ~]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [WARN] 2019-11-07 02:08:42.589 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2019-11-07 02:08:42.599 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.4.1"} [INFO] 2019-11-07 02:08:45.772 [Converge PipelineAction::Create
] Reflections - Reflections took 69 ms to scan 1 urls, producing 20 keys and 40 values [INFO] 2019-11-07 02:08:47.501 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_writer:xxxxxx@172.16.5.199:9200/]}} [WARN] 2019-11-07 02:08:48.808 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://logstash_writer:xxxxxx@172.16.5.199:9200/"} [INFO] 2019-11-07 02:08:49.095 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7} [WARN] 2019-11-07 02:08:49.097 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [INFO] 2019-11-07 02:08:49.173 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://172.16.5.199:9200"]} [WARN] 2019-11-07 02:08:49.309 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [INFO] 2019-11-07 02:08:49.342 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, :thread=>"#"} [INFO] 2019-11-07 02:08:49.370 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template [INFO] 2019-11-07 02:08:49.708 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} warning: thread "Ruby-0-Thread-5: :1" terminated with exception (report_on_exception is true): LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://172.16.5.199:9200/logstash' perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80 perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278 with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373 perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277 Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285 exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:341 rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:359 maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:91 setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:10 setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52 ``` [Next page](https://discuss.elastic.co/t/any-solutions-please/206343.md?page=2)