Hello all,
Setup is
- Logstash (7.7.0) agent on AWS linux instance - using s3 input and Elasticsearch output in confs.
- Elastic Cloud (7.7.0)
Logstash seems to be running and all but then when it comes to seeing the data in ELK (is pointing to the elastic cloud instance), nothing seems to be coming through. Do have other beats on this same instance, e.g. Filebeat and co. and these are all working well and are seeing these in ELK. Am also able to curl to ELK on this instance.
I keep coming across the below "warning" message in the Logstash logs.
{"inflight_count"=>0, "stalling_threads_info"=>{"other"=>[{"thread_id"=>51, "name"=>"[main1]<s3", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/logstash-input-s3-3.5.0/lib/logstash/inputs/s3.rb:237:in `block in push'"}], ["LogStash::Filters::KV", {"add_field"=>{"travelField"=>"To add the destination of travel here"}, "add_tag"=>["travel tag destination"], "id"=>"62fa13960d54c2e68588e31c008f13db25328db5f4689efae5de9636bc013288"}]=>[{"thread_id"=>47, "name"=>"[main1]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:50:in `call'"}, {"thread_id"=>48, "name"=>"[main1]>worker1", "current_call"=>"[...]/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}
and also the below
[logstash.outputs.elasticsearch][main1] Marking url as dead. Last error: [LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://user:xxxxxxxx@xxxxxxxxx.eu-west-2.aws.cloud.es.io:9243/][Manticore::SocketTimeout] Read timed out {:url=>https://user:xxxxxxxx@xxxxxxxxx.eu-west-2.aws.cloud.es.io:9243/, :error_message=>"Elasticsearch Unreachable: [https://user:xxxxxxxx@xxxxxxxxx.eu-west-2.aws.cloud.es.io:9243/][Manticore::SocketTimeout] Read timed out", :error_class=>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError"}
Again I am still able to curl to the Elasticsearch instance and able to get data from other beats that are on this same Logtsash instance.
I am a little stuck as to why data is not able to be seen in ELK. (Note, that is was able to see data previous to 7.7.0). I have seen a couple of these sorts of posts also about Logstash becoming 'Stuck'. but not much has been said. An example is this link here and here.
Would this be maybe due to something within Logstash internally (i.e. something in the s3 plugin gem when on 7.7.0) may be stuck for some reason or something else?