Logstash AWS code 403


I'm trying to push documents from local to elastic server in AWS, and when trying to do so I get 403 error and logstash keeps on trying to establish connection with the server like so:

[2021-05-09T11:09:52,707][TRACE][logstash.inputs.file ][main] Registering file input {:path=>["~/home/ubuntu/json_try/json_try.json"]}
[2021-05-09T11:09:52,737][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x5033269f run>"}
[2021-05-09T11:09:53,441][DEBUG][logstash.outputs.amazonelasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 4s
[2021-05-09T11:09:56,403][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://my-dom.co:8001/scans, :path=>"/"}
[2021-05-09T11:09:56,461][WARN ][logstash.outputs.amazonelasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my-dom.co:8001/scans", :error_type=>LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://my-dom.co:8001/scans/'"}
[2021-05-09T11:09:56,849][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-05-09T11:09:56,853][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2021-05-09T11:09:57,444][DEBUG][logstash.outputs.amazonelasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 8s

I'm using the following logstash conf file:

input {
                type => "json"
                path => "~/home/ubuntu/json_try/json_try.json"
                start_position => "beginning"
                sincedb_path => "/dev/null"

        amazon_es {
                hosts => ["https://my-dom.co/scans"]
                port => 8001
                ssl => true
                region => "us-east-1b"
                index => "snapshot-%{+YYYY.MM.dd}"

Also I've exported AWS keys for the SSL to work.
Is there anything I'm missing in order for the connection to succeed?

It looks like it could be a permission issue. Can you make sure that the logstash user has permission to write/connect to elasticsearch?

I'm specifying ssl as true in the config file and I'm exporting the AWS keys as shown in AWS manual:

Is there anything else needed for the authentication?

You are in the wrong place then. If you are looking for help regarding AWS fork, then you should go here Open Distro for Elasticsearch - Keeping open-source Elasticsearch thriving (opendistrocommunity.dev)


Just to reiterate, we cannot help with the aws service.

You may want to consider upgrading to our Elasticsearch Service - Elastic Cloud: Hosted Elasticsearch, Hosted Search | Elastic

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.