Logstash not pushing data to AWS Elastcisearch

Elastic Stack Logstash
1

Hello everyone,
I am trying to push logs from logstash to Elasticsearch but its giving below error

Sending Logstash's logs to D:/shweta/ELK_poc/logstash-6.3.0/logs which is now configured via log4j2.properties
[2018-06-27T14:06:54,955][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-06-27T14:06:55,752][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.0"}
[2018-06-27T14:06:58,980][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-06-27T14:06:59,824][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/]}}
[2018-06-27T14:06:59,851][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/, :path=>"/"}
[2018-06-27T14:07:00,087][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com:9200/][Manticore::ResolutionFailure] This is usually a temporary error during hostname resolution and means that the local server did not receive a response from an authoritative server (search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com)"}
[2018-06-27T14:07:00,111][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2018-06-27T14:07:00,132][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!", :class=>"ArgumentError", :backtrace=>["D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:31:in read_template_file'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:17:inget_template'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7:in install_template'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:96:ininstall_template'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:26:in register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:97:inregister'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:93:in register'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:340:inregister_plugin'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:351:in register_plugins'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:728:inmaybe_setup_out_plugins'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:361:in start_workers'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:288:inrun'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:248:in block in start'"]} [2018-06-27T14:07:00,151][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com/"]} [2018-06-27T14:07:00,425][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-filter-geoip-5.0.3-java/vendor/GeoLite2-City.mmdb"} [2018-06-27T14:07:01,466][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x474d31fb run>"} [2018-06-27T14:07:01,616][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2018-06-27T14:07:02,250][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2018-06-27T14:07:03,437][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method<' for nil:NilClass>, :backtrace=>["D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:222:in get_event_type'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:47:inevent_action_tuple'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in block in multi_receive'", "org/jruby/RubyArray.java:2486:inmap'", "D:/shweta/ELK_poc/logstash-6.3.0/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:109:inmulti_receive'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:156:in multi_receive'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:475:inblock in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:474:inoutput_batch'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:426:in worker_loop'", "D:/shweta/ELK_poc/logstash-6.3.0/logstash-core/lib/logstash/pipeline.rb:384:inblock in start_workers'"]}
[2018-06-27T14:07:03,843][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

  • Providing other config files in next comment (due to word limit in this one)
2

Continued :

My ES-pipeline.conf file (logstash is configured here)

input {
file {
path => "D:/shweta/ELK_poc/ES-input-logs.txt"
start_position => "beginning"
sincedb_path => "NUL"
ignore_older => 0
}
}

output {
elasticsearch {
hosts => [ "search-test-domain2-2msy6ufh2vl2ztfulhrtoat6hu.us-west-2.es.amazonaws.com/" ]
}
}

  • My log file :

83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-search.png HTTP/1.1" 200 203023 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [04/Jan/2015:05:13:42 +0000] "GET /presentations/logstash-monitorama-2013/images/kibana-dashboard3.png HTTP/1.1" 200 171717 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/highlight/highlight.js HTTP/1.1" 200 26185 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [04/Jan/2015:05:13:44 +0000] "GET /presentations/logstash-monitorama-2013/plugin/zoom-js/zoom.js HTTP/1.1" 200 7697 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"
83.149.9.216 - - [04/Jan/2015:05:13:45 +0000] "GET /presentations/logstash-monitorama-2013/plugin/notes/notes.js HTTP/1.1" 200 2892 "http://semicomplete.com/presentations/logstash-monitorama-2013/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.77 Safari/537.36"

Can anyone tell me whats wrong here.

3

I think you need to use a special amazon_es output plugin.

4

Can you provide its details , how can i use this plugin or what command to be used , or where to get this plugin from

Thanks in advance

5

I have never used it as the standard Elasticsearch output plugin works with our Elastic Cloud service. You should however be able to install it like this: logstash-plugin install logstash-output-amazon_es

Documentation seems to be available in the GitHub repository.

6

Tried installing this plugin but giving this error :

D:\shweta\ELK_poc\logstash-6.3.0\bin>logstash-plugin install logstash-output-amazon_es
Validating logstash-output-amazon_es
Unable to download data from https://rubygems.org - SocketError: Failed to open TCP connection to rubygems.org:443 (initialize: name or service not known) (https://rubygems.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for logstash-output-amazon_es

I think its firewall issue of my company. So i manually downloaded the plugin using gitclone. How can i install it now.

And as you said that standard Elasticsearch output plugin works with Elastic cloud , then why its isnt working in my case.
Could you help

Thanks

7

As Elastic Cloud supports role-based access controls, you can connect using HTTPS and basic auth, which the standard Elasticsearch plugin supports. AWS Elasticsearch service does not support this and therefore require a special plugin.

If you need help configuring the amazon_es plugin, I would recommend you contact Amazon support as they are maintaining that plugin.

8

Ok .
Thanks for the info Christian :slight_smile:
Need to figure out now how to make it work

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.