Hi,
I config AWS ES for the elasticsearch service. I can use curl or python to load single doc to it without any problem. However, when I tried to upload data through logstash (I use docker image from another EC2), I always got mainly two kinds of errors:
first, the healthy check error, because of can't connect to port 9200, but I would assume this error won't affect function
second, An unexpected error occurred! {:error=>#<NoMethodError: undefined method `<' for nil:NilClass>,...... (you can get the detailed information from below log).
I use latest ES and logstash. Below is my logstash conf:
input { stdin { } }
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => ["search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com"]
manage_template => false
index => "logstash-%{+YYYY.MM.dd}"
ssl => true
}
stdout { codec => rubydebug }
}
Even I installed the plugin "logstash-output-amazon_es", I got the same error.
I would greatly appreciate your help!
Thanks,
Sam
bash-4.2$ logstash -f ./conf/command.conf
Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2018-04-12T06:12:44,394][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-04-12T06:12:44,415][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-04-12T06:12:45,040][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-12T06:12:45,671][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-12T06:12:46,029][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-12T06:12:49,982][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-12T06:12:50,554][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/]}}
[2018-04-12T06:12:50,565][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/, :path=>"/"}
[2018-04-12T06:13:10,791][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-04-12T06:13:10,811][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com"]}
[2018-04-12T06:13:11,082][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2d14303b run>"}
The stdin plugin is now waiting for input:
[2018-04-12T06:13:11,190][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
[2018-04-12T06:13:15,805][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/, :path=>"/"}
[2018-04-12T06:13:35,834][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/][Manticore::ConnectTimeout] connect timed out"}
[2018-04-12T06:13:36,835][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-es-5-5-wxzz2y5456vntppkwwkntfe7b4.us-east-1.es.amazonaws.com:9200/, :path=>"/"}
127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
[2018-04-12T06:13:49,973][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method <' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:213:inget_event_type'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:165:in event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:39:inevent_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in block in multi_receive'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.3-java/lib/logstash/outputs/elasticsearch/common.rb:34:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:inmulti_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:49:in multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:479:inblock in output_batch'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:478:inoutput_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:430:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:385:inblock in start_workers'"]}
[2018-04-12T06:13:50,131][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
bash-4.2$