Error while connecting to Amazon ES using Logstash on EC2 [403 error]

I am connecting to AWS ES using logstash installed on AWS EC2. I have this issue since 2 days and not able to resolve. Any help would be appreciated.
Below is the error log:

[root@ip-10-0-0-7 logstash]# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/toecs.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-12-16 19:35:10.019 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-12-16 19:35:10.033 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.5"}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:33: warning: already initialized constant ROOT_URI_PATH
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:36: warning: already initialized constant DEFAULT_OPTIONS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:160: warning: already initialized constant ES1_SNIFF_RE_URL
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:7: warning: already initialized constant DEFAULT_HEADERS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/http_client.rb:24: warning: already initialized constant TARGET_BULK_BYTES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:8: warning: already initialized constant DOC_DLQ_CODES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:9: warning: already initialized constant DOC_SUCCESS_CODES
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:10: warning: already initialized constant DOC_CONFLICT_CODE
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:16: warning: already initialized constant VERSION_TYPES_PERMITTING_CONFLICT
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:133: warning: already initialized constant VALID_HTTP_ACTIONS
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:247: warning: already initialized constant DEFAULT_EVENT_TYPE_ES6
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0-java/lib/logstash/outputs/amazon_es/common.rb:248: warning: already initialized constant DEFAULT_EVENT_TYPE_ES7
[INFO ] 2019-12-16 19:35:17.194 [Converge PipelineAction::Create] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
url template
{:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
[INFO ] 2019-12-16 19:35:17.687 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/]}}
[INFO ] 2019-12-16 19:35:17.696 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:18.356 [[main]-pipeline-manager] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
[INFO ] 2019-12-16 19:35:18.385 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com"]}
[INFO ] 2019-12-16 19:35:18.485 [Converge PipelineAction::Create] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b733c8c sleep>"}
The stdin plugin is now waiting for input:
[INFO ] 2019-12-16 19:35:18.539 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2019-12-16 19:35:18.787 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
[INFO ] 2019-12-16 19:35:23.389 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:23.412 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
[INFO ] 2019-12-16 19:35:28.414 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:28.427 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}
^C[WARN ] 2019-12-16 19:35:33.316 [SIGINT handler] runner - SIGINT received. Shutting down.
[INFO ] 2019-12-16 19:35:33.429 [Ruby-0-Thread-4: :1] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-12-16 19:35:33.445 [Ruby-0-Thread-4: :1] elasticsearch - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://search-elk1214-rsauprx2rc5v5xjm57ev6agh5a.us-east-2.es.amazonaws.com:443/'"}

this seems to be a permission issue. Can you make sure that the logstash user has permission to write/connect to elasticsearch?

Thank you. Yes it was permission issue and got it resolved.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.