I am running logstash on RHEL with amazon_es output plugin to send data to aws elastic search but getting this output and i don't see index in kibana. how do i make sure everything is working fine?
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-02 05:51:34.547 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-02 05:51:34.561 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.7.1"}
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:33: warning: already initialized constant ROOT_URI_PATH
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:36: warning: already initialized constant DEFAULT_OPTIONS
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/pool.rb:160: warning: already initialized constant ES1_SNIFF_RE_URL
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:7: warning: already initialized constant DEFAULT_HEADERS
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/http_client.rb:24: warning: already initialized constant TARGET_BULK_BYTES
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:8: warning: already initialized constant DOC_DLQ_CODES
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:9: warning: already initialized constant DOC_SUCCESS_CODES
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:10: warning: already initialized constant DOC_CONFLICT_CODE
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:16: warning: already initialized constant VERSION_TYPES_PERMITTING_CONFLICT
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:133: warning: already initialized constant VALID_HTTP_ACTIONS
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:247: warning: already initialized constant DEFAULT_EVENT_TYPE_ES6
/usr/share/logstash/logstash-output-amazon_es/lib/logstash/outputs/amazon_es/common.rb:248: warning: already initialized constant DEFAULT_EVENT_TYPE_ES7
[INFO ] 2019-05-02 05:51:41.492 [Converge PipelineAction::Create] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
url template
{:scheme=>nil, :user=>nil, :password=>nil, :host=>"URLTEMPLATE", :port=>443, :path=>nil}
[INFO ] 2019-05-02 05:51:41.971 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[https://search-yhsb-test-pwnkrhvj5ct4ikyldwungz5u6a.eu-west-1.es.amazonaws.com:443/]}}
[INFO ] 2019-05-02 05:51:41.979 [[main]-pipeline-manager] elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://search-yhsb-test-pwnkrhvj5ct4ikyldwungz5u6a.eu-west-1.es.amazonaws.com:443/, :path=>"/"}
[WARN ] 2019-05-02 05:51:42.524 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"https://search-yhsb-test-pwnkrhvj5ct4ikyldwungz5u6a.eu-west-1.es.amazonaws.com:443/"}
[INFO ] 2019-05-02 05:51:42.708 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-05-02 05:51:42.712 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-05-02 05:51:42.740 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//search-yhsb-test-pwnkrhvj5ct4ikyldwungz5u6a.eu-west-1.es.amazonaws.com"]}
[INFO ] 2019-05-02 05:51:42.758 [Ruby-0-Thread-5: :1] elasticsearch - Using mapping template from {:path=>nil}
[INFO ] 2019-05-02 05:51:42.777 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2019-05-02 05:51:43.127 [Converge PipelineAction::Create] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x591b025b run>"}
[INFO ] 2019-05-02 05:51:43.193 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[INFO ] 2019-05-02 05:51:43.202 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2019-05-02 05:51:43.586 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
My output file content is.
output {
amazon_es {
hosts => ["search-yhsb-test-pwnkrhvj5ct4ikyldwungz5u6a.eu-west-1.es.amazonaws.com" ]
region => "eu-west-1"
index => "logstash-aperture-iPPW2-devo-na-eu-service"
aws_access_key_id => 'DLURO3TJNQCD4B'
aws_secret_access_key => '0wenj+O+4JAArlMJMKwz9H1'
}
}