Logstash not pushing data to AWS Elasticsearch endpoint

[Shweta_Priyadarshani]

Hi all,
I am trying to push my logs from logstash to elasticsearch but its failing. here is my logstash.conf file :

        input {
        		file {
        				path => "D:/shweta/ELK_poc/test3.txt"
        				start_position => "beginning"
        				sincedb_path => "NUL"
        				ignore_older => 0
        			}}

        output {
            elasticsearch {
                hosts => [ "https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com" ]
        		index => "testindex4-5july"
        		document_type => "test-file"
            }
        } 

The ES endpoint that i have provided in hosts is open , so there should not be an access isssue, but it still gives following error:

_[2018-07-05T13:59:05,753][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/, :path=>"/"}_
_[2018-07-05T13:59:05,769][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/][Manticore::ResolutionFailure] This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server (search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com)"}_

I am stuck here. But when i downloaded ES and installed it in my machine and ran it locally , with the following logstash.conf file , it worked all good pushing data to local es

input {
		file {				
                                path => "D:/shweta/ELK_poc/AEM-error-logs.log"
				start_position => "beginning"
				sincedb_path => "NUL"
				ignore_older => 0
			}
	  }
	  
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

I tried a lot of ways but not able to resolve the issue , can anyone please help. I don't want to give localhost but AWS ES domain endpoint. Any hints or leads will be highly appreciated

Thanks in advance
Shweta

[dadoonet (David Pilato)]

I moved your question to #logstash.

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

This is the icon to use if you are not using markdown format:

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is the only way to have access to X-Pack. Think about what is there yet like Security, Monitoring, Reporting and what is coming like Canvas, SQL...

[Christian_Dahlqvist]

You need to use the the custom amazon_es output plugin.