I've configured the ES in AWS in elasticsearch service, and logstash is configured separately in another server. All my production servers are pushing logs to logstash via filebeat services. This system was working end to end correctly so far. But in a recently noticed those logs which are collected by logstash are not forward to ES and kibana is not showing any logs on that time period.
This happens intermittently, and still I was not able to find any clue though out logstash-stdout.log, but followings on the logstash-stderr.log
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client.rb:24: warning: already initialized constant TARGET_BULK_BYTES
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:8: warning: already initialized constant DOC_DLQ_CODES
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:9: warning: already initialized constant DOC_SUCCESS_CODES
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:10: warning: already initialized constant DOC_CONFLICT_CODE
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:16: warning: already initialized constant VERSION_TYPES_PERMITTING_CONFLICT
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:133: warning: already initialized constant VALID_HTTP_ACTIONS
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:247: warning: already initialized constant DEFAULT_EVENT_TYPE_ES6
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/common.rb:248: warning: already initialized constant DEFAULT_EVENT_TYPE_ES7
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:33: warning: already initialized constant ROOT_URI_PATH
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:36: warning: already initialized constant DEFAULT_OPTIONS
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:160: warning: already initialized constant ES1_SNIFF_RE_URL
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/pool.rb:161: warning: already initialized constant ES2_SNIFF_RE_URL
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-amazon_es-6.4.0-java/lib/logstash/outputs/amazon_es/http_client/manticore_adapter.rb:7: warning: already initialized constant DEFAULT_HEADERS
Further tested in the all logstash configurations and filters and found no issues.
But after 7-8 hours it automatically pushes logs to ES host and visible in Kibana, this happed for me 3-4 times within a month.
What would be the reason for this issue ? or anyone got same issues ?