Error while connecting to Amazon Elasticsearch

Elastic Stack Logstash
1

Hi,

I am a beginner and trying to figure our ELK. I have logstash configured in an AWS EC2 instance. I am trying to transfer the data from S3 bucket to AWS ES service using logstash. I am getting the following error when I run logstash.

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-05-13 03:23:25.868 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-13 03:23:25.880 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.6.2"}
[INFO ] 2020-05-13 03:23:27.795 [Converge PipelineAction::Create<main>] Reflections - Reflections took 57 ms to scan 1 urls, producing 20 keys and 40 values
url template
{:scheme=>"https", :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
[INFO ] 2020-05-13 03:23:40.351 [[main]-pipeline-manager] amazonelasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://search-cyberdemo-xxxxxxxxxxxxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com:443/]}}
[INFO ] 2020-05-13 03:23:40.360 [[main]-pipeline-manager] amazonelasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-cyberdemo-xxxxxxxxxxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com:443/, :path=>"/"}
[ERROR] 2020-05-13 03:23:40.479 [[main]-pipeline-manager] javapipeline - Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NoMethodError: undefined method `credentials' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/signers/v4.rb:45:in `initialize'", 
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:108:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:291:in `perform_request_to_url'",
"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:245:in `block in healthcheck!'", "org/jruby/RubyHash.java:1428:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:241:in `healthcheck!'",
 "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:341:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:71:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:304:in `build_pool'", 
 "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client.rb:54:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:110:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/http_client_builder.rb:98:in `build'", 
 "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es.rb:256:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-amazon_es-7.0.1-java/lib/logstash/outputs/amazon_es/common.rb:25:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:200:in `block in register_plugins'", 
 "org/jruby/RubyArray.java:1814:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:199:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:501:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:212:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:154:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:109:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/s3_input.conf"], :thread=>"#<Thread:0x53b08df6 run>"}
[ERROR] 2020-05-13 03:23:40.493 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[INFO ] 2020-05-13 03:23:40.756 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-13 03:23:45.635 [LogStash::Runner] runner - Logstash shut down.

Below is my config file

input {
  s3 {
     access_key_id => ""
     secret_access_key => ""
     region => "us-east-2"
     bucket => "s3-XXXXX"
     prefix => "logs"
     #interval => "10"
     #additional_settings => {
      #force_path_style => true
      #follow_redirects => false}
  }
}
 
output {
	amazon_es {
    hosts => ["https://search-cyberdemo-xxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com"]
	region => "us-east-2"
    index => "logs-%{+YYYY.MM.dd}"
	ssl => true
	    
  }
}

Thanks in advance

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.