Flushing buffer at interval

Elastic Stack Logstash
1

I am curious if someone can assist me. It appears there's an issue with either the latest Logstash or perhaps Elasticsearch can't handle the traffic, which I don't find to be true. I am seeing the logs being fed into Elasticsearch, but after a while...it seems to no longer want to work. This, I believe, started to happen after Logstash was upgraded on my ELK server. I am not 100% sure...just a theory. We had been using Logstash version 2.2.3 and everything had been working great for at least a month or two before things started to not work.

Logstash: 2.3.1
Elasticsearch: 2.3.4
Redis: 3.2.1

ELK Server Specs:
CPU: 40 processors @ 10 cores each
RAM: 125G
DISK: 1.1T (61% Used)
So I don't think my server is the issue.

Here's how it's sending traffic:

It's a pretty standard layout. I have my two sensors sending the BRO logs using Logstash to Redis on the ELK server. Logstash on the ELK server grabs those logs and sends them to Elasticsearch.

Redis shows that it's queuing up the traffic for both the "keys" that each sensor is feeding it, and ready for Logstash to send it to Elasticsearch. I can sometimes restart Logstash on the ELK server and it'll start to process the queue from Redis, but then after a while...it'll stop and give the following message:

Flush Buffer Message.PNG1260×112 10.6 KB

It'll then stop and continue to display those messages for a little while. If I let it sit there, throughout the day it'll continue to process more data before it stops and displays the same message.

I am not sure if Java is the issue or perhaps the change in Logstash/Elasticsearch caused this. Anyone have any suggestions?

2

Please don't post pictures of text, they are difficult to read and some people may not be even able to see them.

Can you repost the (formatted) text error?

3

This is the message. The 'at-symbol' is actually @ but it wouldn't let me post it, which is why I used the picture.

Flushing buffer at interval {:instance=>"#<LogStash::Outputs::ElasticSearch::Buffer:0x7e195a6f 'at-symbol'operations_mutex=#Mutex:0x70debbdc, 'at-symbol'max_size=500, 'at-symbol'operations_lock=#Java::JavaUtilConcurrentLocks::ReentrantLock:0x74e1311f, 'at-symbol'submit_proc=#Proc:0x51d5f96'at-symbol'/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:57, 'at-symbol'logger=#<Cabin::Channel:0x9804b2 'at-symbol'metrics=#<Cabin::Metrics:0x7d8cb477 'at-symbol'metrics_lock=#Mutex:0x51cc4ea8, 'at-symbol'metrics={}, 'at-symbol'channel=#<Cabin::Channel:0x9804b2 ...>>, 'at-symbol'subscriber_lock=#Mutex:0x3ec25ae8, 'at-symbol'level=:debug, 'at-symbol'subscribers={12642=>#<Cabin::Subscriber:0x668858aa 'at-symbol'output=#<Cabin::Outputs::IO:0x5582f3dd 'at-symbol'io=#<IO:fd 1>, 'at-symbol'lock=#Mutex:0x79bbf69f>, 'at-symbol'options={}>}, 'at-symbol'data={}>, 'at-symbol'last_flush=2016-07-27 08:51:57 -0400, 'at-symbol'flush_interval=1, 'at-symbol'stopping=#Concurrent::AtomicBoolean:0x7ca1a27a, 'at-symbol'buffer=[], 'at-symbol'flush_thread=#<Thread:0x5848f84c run>>", :interval=>1, :level=>:debug, :file=>"logstash/outputs/elasticsearch/buffer.rb", :line=>"90", :method=>"interval_flush"}

4

Use code formatting, the </> button, or back ticks ```.

5

Does anyone have a solution to this issue? I have almost the exact same issue.