# Logstash not pushing data to AWS Elastcisearch **URL:** \ **Category:** Logstash\ **Created:** [June 27, 2018, 9:08am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579 "2018-06-27T09:08:28Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Shweta\_Priyadarshani](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Post date:** [June 27, 2018, 9:08am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/1 "2018-06-27T09:08:28Z")
Hello everyone, I am trying to push logs from logstash to Elasticsearch but its giving below error Sending Logstash's logs to D:/shweta/ELK\_poc/logstash-6.3.0/logs which is now configured via log4j2.properties [2018-06-27T14:06:54,955][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2018-06-27T14:06:55,752][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.3.0"} [2018-06-27T14:06:58,980][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>2, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} [2018-06-27T14:06:59,824][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/](https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/)]}} [2018-06-27T14:06:59,851][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/](https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/), :path=\>"/"} [2018-06-27T14:07:00,087][WARN][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=\>"[https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/](https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/)", :error\_type=\>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=\>"Elasticsearch Unreachable: [[https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/](https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/)][Manticore::ResolutionFailure] This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server ([search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com](http://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com))"} [2018-06-27T14:07:00,111][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} [2018-06-27T14:07:00,132][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=\>"Template file '' could not be found!", :class=\>"ArgumentError", :backtrace=\>["D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template\_manager.rb:31:in `read_template_file'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:in`get\_template'", "D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template\_manager.rb:7:in `install_template'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:in`install\_template'", "D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:97:in`register'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:93:in `register'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:340:in`register\_plugin'", "D:/shweta/ELK\_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in`each'", "D:/shweta/ELK\_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in `register_plugins'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:728:in`maybe\_setup\_out\_plugins'", "D:/shweta/ELK\_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:361:in `start_workers'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:288:in`run'", "D:/shweta/ELK\_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"]} [2018-06-27T14:07:00,151][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com/"]} [2018-06-27T14:07:00,425][INFO][logstash.filters.geoip] Using geoip database {:path=>"D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"} [2018-06-27T14:07:01,466][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"} [2018-06-27T14:07:01,616][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2018-06-27T14:07:02,250][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} [2018-06-27T14:07:03,437][FATAL][logstash.runner] An unexpected error occurred! {:error=>#, :backtrace=\>["D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:222:in `get_event_type'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:47:in`event\_action\_tuple'", "D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in`map'", "D:/shweta/ELK\_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:109:in`multi\_receive'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:156:in `multi_receive'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:475:in`block in output\_batch'", "org/jruby/RubyHash.java:1343:in `each'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:474:in`output\_batch'", "D:/shweta/ELK\_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:426:in `worker_loop'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:384:in`block in start\_workers'"]} [2018-06-27T14:07:03,843][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit - Providing other config files in next comment (due to word limit in this one) ---
**Author:** ![Shweta\_Priyadarshani](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Post date:** [June 27, 2018, 9:09am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/2 "2018-06-27T09:09:27Z")
Continued : My ES-pipeline.conf file (logstash is configured here) input { file { path =\> "D:/shweta/ELK\_poc/ES-input-logs.txt" start\_position =\> "beginning" sincedb\_path =\> "NUL" ignore\_older =\> 0 } } output { elasticsearch { hosts =\> ["[search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com/](http://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com/)" ] } } - My log file : 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" 83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "[http://semicomplete.com/presentations/logstash-monitorama-2013/](http://semicomplete.com/presentations/logstash-monitorama-2013/)" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36" Can anyone tell me whats wrong here. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 27, 2018, 9:16am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/3 "2018-06-27T09:16:26Z")
I think you need to use a special `amazon_es` output plugin. ---
**Author:** ![Shweta\_Priyadarshani](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Post date:** [June 27, 2018, 9:17am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/4 "2018-06-27T09:17:39Z")
Can you provide its details , how can i use this plugin or what command to be used , or where to get this plugin from Thanks in advance ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 27, 2018, 9:22am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/5 "2018-06-27T09:22:43Z")
I have never used it as the standard Elasticsearch output plugin works with our [Elastic Cloud service](https://www.elastic.co/cloud). You should however be able to install it like this: `logstash-plugin install logstash-output-amazon_es` Documentation seems to be available [in the GitHub repository](https://github.com/awslabs/logstash-output-amazon_es). ---
**Author:** ![Shweta\_Priyadarshani](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Post date:** [June 27, 2018, 9:31am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/6 "2018-06-27T09:31:59Z")
Tried installing this plugin but giving this error : D:\shweta\ELK\_poc\logstash-6.3.0\bin\>logstash-plugin install logstash-output-amazon\_es Validating logstash-output-amazon\_es Unable to download data from [https://rubygems.org](https://rubygems.org) - SocketError: Failed to open TCP connection to [rubygems.org:443](http://rubygems.org:443) (initialize: name or service not known) ([https://rubygems.org/latest\_specs.4.8.gz](https://rubygems.org/latest_specs.4.8.gz)) ERROR: Installation aborted, verification failed for logstash-output-amazon\_es I think its firewall issue of my company. So i manually downloaded the plugin using gitclone. How can i install it now. And as you said that standard Elasticsearch output plugin works with Elastic cloud , then why its isnt working in my case. Could you help Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 27, 2018, 9:42am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/7 "2018-06-27T09:42:20Z")
As Elastic Cloud supports role-based access controls, you can connect using HTTPS and basic auth, which the standard Elasticsearch plugin supports. AWS Elasticsearch service does not support this and therefore require a special plugin. If you need help configuring the amazon\_es plugin, I would recommend you contact Amazon support as they are maintaining that plugin. ---
**Author:** ![Shweta\_Priyadarshani](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@Shweta\_Priyadarshani](https://discuss.elastic.co/u/Shweta_Priyadarshani)\ **Post date:** [June 27, 2018, 10:19am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/8 "2018-06-27T10:19:45Z")
Ok . Thanks for the info Christian 🙂 Need to figure out now how to make it work ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 25, 2018, 10:19am UTC](https://discuss.elastic.co/t/logstash-not-pushing-data-to-aws-elastcisearch/137579/9 "2018-07-25T10:19:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.