Logstash with Redis

Elastic Stack Logstash
1

I have an ELK stack with redis as my queue. Currently I am running one redis instance on my sensors and it reports back to a central redis "cluster". I put cluster in quotes because I don't have it set up as a cluster rather I have 10 VM's running redis and placed them in their own domain. My issues is that all my nodes are receiving data from the sensors sending it but not all of them are sending their data to ES. Some of the redis nodes just hold the data and never BLOP it into the cluster. my current configuration for logstash-redis is:

input {
redis {
host => [ "192.168.99.4" ]
data_type => "list"
key => "bro"
}
}
output {
if [type] == "BRO" {
redis {
host => [ "redis-op-01-cl-55" ]
data_type => "list"
key => "bro"
}
}
}

My question is: Is there an issue using domain names in the input/output sections of logstash and could that be causing my problem? Like I said previously it appears all my redis nodes are receiving data correctly but not the data on some nodes do not get pulled off to the cluster. I am perplexed and would appreciate any help.

Also I am seeing the following error:
{:timestamp=>"2015-09-16T12:55:00.775000-0400", :message=>"retrying failed action with response code: 429", :level=>:warn}

Thanks

2

You could try increasing logging to see what is happening?

3

upon further investigation the error from logstash I am getting is:

{:timestamp=>"2015-10-08T09:18:59.289000-0400", :message=>"Failed to send event to Redis", :event=>#<LogStash::Event:0x42c3e5fc @metadata_accessors=#<LogStash::Util::Accessors:0x197c0ecf @store={}, @lut={}>, @cancelled=false, @data={"ts"=>"2015-10-08T12:33:30.092384Z", "uid"=>"C5by8c4qD4NkvK0U96", "id.orig_h"=>"192.168.1.253", "id.orig_p"=>63203, "id.resp_h"=>"8.8.8.8", "id.resp_p"=>53, "proto"=>"udp", "service"=>"dns", "duration"=>0.032474, "orig_bytes"=>102, "resp_bytes"=>224, "conn_state"=>"SF", "local_orig"=>true, "local_resp"=>false, "missed_bytes"=>0, "history"=>"Dd", "orig_pkts"=>2, "orig_ip_bytes"=>158, "resp_pkts"=>2, "resp_ip_bytes"=>280, "tunnel_parents"=>#Java::JavaUtil::ArrayList:0x502c830, "resp_cc"=>"US", "sensorname"=>"satcon99-eth2", "@version"=>"1", "@timestamp"=>"2015-10-08T12:33:30.092Z", "host"=>"satcon99", "path"=>"/nsm/bro/logs/current/conn.log", "type"=>"BRO", "csp_sensor"=>"cobalt", "_path"=>"conn"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x451128d @store={"ts"=>"2015-10-08T12:33:30.092384Z", "uid"=>"C5by8c4qD4NkvK0U96", "id.orig_h"=>"192.168.1.253", "id.orig_p"=>63203, "id.resp_h"=>"8.8.8.8", "id.resp_p"=>53, "proto"=>"udp", "service"=>"dns", "duration"=>0.032474, "orig_bytes"=>102, "resp_bytes"=>224, "conn_state"=>"SF", "local_orig"=>true, "local_resp"=>false, "missed_bytes"=>0, "history"=>"Dd", "orig_pkts"=>2, "orig_ip_bytes"=>158, "resp_pkts"=>2, "resp_ip_bytes"=>280, "tunnel_parents"=>#Java::JavaUtil::ArrayList:0x502c830, "resp_cc"=>"US", "sensorname"=>"satcon99-eth2", "@version"=>"1", "@timestamp"=>"2015-10-08T12:33:30.092Z", "host"=>"satcon99", "path"=>"/nsm/bro/logs/current/conn.log", "type"=>"BRO", "csp_sensor"=>"cobalt", "_path"=>"conn"}, @lut={"[type]"=>[{"ts"=>"2015-10-08T12:33:30.092384Z", "uid"=>"C5by8c4qD4NkvK0U96", "id.orig_h"=>"192.168.1.253", "id.orig_p"=>63203, "id.resp_h"=>"8.8.8.8", "id.resp_p"=>53, "proto"=>"udp", "service"=>"dns", "duration"=>0.032474, "orig_bytes"=>102, "resp_bytes"=>224, "conn_state"=>"SF", "local_orig"=>true, "local_resp"=>false, "missed_bytes"=>0, "history"=>"Dd", "orig_pkts"=>2, "orig_ip_bytes"=>158, "resp_pkts"=>2, "resp_ip_bytes"=>280, "tunnel_parents"=>#Java::JavaUtil::ArrayList:0x502c830, "resp_cc"=>"US", "sensorname"=>"satcon99-eth2", "@version"=>"1", "@timestamp"=>"2015-10-08T12:33:30.092Z", "host"=>"satcon99", "path"=>"/nsm/bro/logs/current/conn.log", "type"=>"BRO", "csp_sensor"=>"cobalt", "_path"=>"conn"}, "type"]}>>, :identity=>"default", :exception=>#<Redis::CannotConnectError: Error connecting to Redis on redis-op-01-cl-55:6379 (Errno::ECONNREFUSED)>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:331:in establish_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:323:inestablish_connection'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:94:in connect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:279:inwith_reconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:277:in with_reconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:93:inconnect'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:350:in ensure_connected'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:207:inprocess'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:292:in logging'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:206:inprocess'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis/client.rb:112:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis.rb:997:inrpush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis.rb:37:in synchronize'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/monitor.rb:211:inmon_synchronize'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/monitor.rb:210:in mon_synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis.rb:37:insynchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/redis-3.2.1/lib/redis.rb:996:in rpush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-redis-0.1.4/lib/logstash/outputs/redis.rb:169:inreceive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/outputs/base.rb:88:in handle'", "(eval):31:inoutput_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:244:in outputworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.0-java/lib/logstash/pipeline.rb:166:instart_outputs'"], :level=>:warn}

I am not sure why I see " <Redis::CannotConnectError: Error connecting to Redis on redis-op-01-cl-55:6379 (Errno::ECONNREFUSED)>".

4

My config on this redis node is the same as it is on all my other redis nodes. I am able to connect to the domain redis-op-01-cl-55. Is there an error in logstash where instead of having the domain in the config I need to list out the IP addresses in an array? I thought that :
output {
if [type] == "BRO" {
redis {
host => [ "redis-op-01-cl-55" ]
data_type => "list"
key => "bro"
}
}
would the output to the domain not create an array from all the IP addressess hosted on that domain? Should I just list them out individually?

5

No, it could have been transitory, unless it's still happening.
You can always try an IP.