Logstash does not output to elasticsearch, only "Flushing buffer at interval" in log

Elastic Stack Logstash
1

My logstash setup stopped outputting to Elasticsearch earlier today. When I run with --debug option, I don't see any error but only mostly the lines like the following:

{:timestamp=>"2016-02-10T03:29:43.431000+0000", :message=>"Flushing buffer at interval", :instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x184f3551 @operations_mutex=#<Mutex:0x1688d828>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x12b75e21>, @submit_proc=#<Proc:0x36b18009@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:55>, @logger=#<Cabin::Channel:0x66a8476f @metrics=#<Cabin::Metrics:0x4e464bcd @metrics_lock=#<Mutex:0x379a2d5c>, @metrics={}, @channel=#<Cabin::Channel:0x66a8476f ...>>, @subscriber_lock=#<Mutex:0xd11426e>, @level=:debug, @subscribers={12590=>#<Cabin::Outputs::IO:0x5542c156 @io=#<File:/data/debug.log>, @lock=#<Mutex:0x33d4f66f>>}, @data={}>, @last_flush=2016-02-10 03:29:42 +0000, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x34d1c049>, @buffer=[], @flush_thread=#<Thread:0x2a150350 run>>", :interval=>1, :level=>:info, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}
{:timestamp=>"2016-02-10T03:29:44.433000+0000", :message=>"Flushing buffer at interval", :instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x184f3551 @operations_mutex=#<Mutex:0x1688d828>, @max_size=500, @operations_lock=#<Java::JavaUtilConcurrentLocks::ReentrantLock:0x12b75e21>, @submit_proc=#<Proc:0x36b18009@/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:55>, @logger=#<Cabin::Channel:0x66a8476f @metrics=#<Cabin::Metrics:0x4e464bcd @metrics_lock=#<Mutex:0x379a2d5c>, @metrics={}, @channel=#<Cabin::Channel:0x66a8476f ...>>, @subscriber_lock=#<Mutex:0xd11426e>, @level=:debug, @subscribers={12590=>#<Cabin::Outputs::IO:0x5542c156 @io=#<File:/data/debug.log>, @lock=#<Mutex:0x33d4f66f>>}, @data={}>, @last_flush=2016-02-10 03:29:43 +0000, @flush_interval=1, @stopping=#<Concurrent::AtomicBoolean:0x34d1c049>, @buffer=[], @flush_thread=#<Thread:0x2a150350 run>>", :interval=>1, :level=>:info, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}

I confirm that Elasticsearch cluster that my logstash setup is talking to is up and running with green status and that there are no logstash-forwarder error in any of the machines that piped logs to my logstash setup.

Any help in how to debug and fix this issue would be greatly appreciated.

2

It'd help if you provided your config :slight_smile:

3

Thanks @warkolm. It turned out to be an underlying networking issue on my logstash host. After I reboot the machine, logstash is back up and operational. I guess I should have tried rebooting first :slightly_smiling:. Thanks again.

4

Can you please provide a detailed explanation of your solution. I am getting the exact same error. What type of networking issue it was? How to reboot the machine?

5

sure, basically, I just literally power off the machine and power back on and the problem went away. I found out later that some other engineer has been trying to run logstash and elasticsearch as docker containers in the same machine so that might have caused the original issue.

6

Can you please provide a detailed explanation of your solution. I am getting the exact same error. What type of networking issue it was? How to reboot the machine?

7

rebooted the server, but getting the same issue