I am connecting to AWS ES using logstash installed on AWS EC2. I have this issue since 2 days and not able to resolve. Any help would be appreciated.
Below is the error log:
[root@ip-10-0-0-7 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/toecs.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-12-16 19:35:10.019 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-12-16 19:35:10.033 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.5"}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:33: warning: already initialized constant ROOT_URI_PATH
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:36: warning: already initialized constant DEFAULT_OPTIONS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:160: warning: already initialized constant ES1_SNIFF_RE_URL
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:7: warning: already initialized constant DEFAULT_HEADERS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client.rb:24: warning: already initialized constant TARGET_BULK_BYTES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:8: warning: already initialized constant DOC_DLQ_CODES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:9: warning: already initialized constant DOC_SUCCESS_CODES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:10: warning: already initialized constant DOC_CONFLICT_CODE
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:16: warning: already initialized constant VERSION_TYPES_PERMITTING_CONFLICT
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:133: warning: already initialized constant VALID_HTTP_ACTIONS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:247: warning: already initialized constant DEFAULT_EVENT_TYPE_ES6
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:248: warning: already initialized constant DEFAULT_EVENT_TYPE_ES7
[INFO ] 2019-12-16 19:35:17.194 [Converge PipelineAction::Create] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
url template
{:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
[INFO ] 2019-12-16 19:35:17.687 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/]}}
[INFO ] 2019-12-16 19:35:17.696 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:18.356 [[main]-pipeline-manager] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
[INFO ] 2019-12-16 19:35:18.385 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com"]}
[INFO ] 2019-12-16 19:35:18.485 [Converge PipelineAction::Create] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b733c8c sleep>"}
The stdin plugin is now waiting for input:
[INFO ] 2019-12-16 19:35:18.539 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2019-12-16 19:35:18.787 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
[INFO ] 2019-12-16 19:35:23.389 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:23.412 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
[INFO ] 2019-12-16 19:35:28.414 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:28.427 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
^C[WARN ] 2019-12-16 19:35:33.316 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2019-12-16 19:35:33.429 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:33.445 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}