Topbeat - unreachable redis blocks logstash output

[rajamadhavan]

Hi,
For a specific requirement, I have to send topbeat data to 2 different outputs simultaneously

1. Redis
2. logstash

Under normal cases when both are reachable, data reaches both outputs without any issue.

If logstash host is unreachable and redis host is reachable, I see the data being received at the redis, though topbeat tries to reconnect to logstash periodically.

However, if the redis host is unreachable and logstash host is reachable, topbeat is blocked retrying connection to redis and data is not received on the logstash. I tried playing around with reconnect_interval and timeout setting(high and low) with no effect.

Is there a workaround or configuration option to avoid blocking ? As I said, this is a very specific requirement and hence the option of using logstash to output using redis output plugin is not what I need.

topbeat version used - 1.3.1

Output on topbeat start

raja@raja-Inspiron-3542:~$ sudo topbeat -c /etc/topbeat/topbeat.yml -e -d "*"
2017/04/17 08:35:17.424181 beat.go:156: DBG  Initializing output plugins
2017/04/17 08:35:17.424223 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2017/04/17 08:35:17.424564 logstash.go:106: INFO Max Retries set to: 3
2017/04/17 08:35:17.424573 client.go:100: DBG  connect
2017/04/17 08:35:17.424933 outputs.go:126: INFO Activated logstash as output plugin.
2017/04/17 08:35:17.424964 redis.go:95: INFO Reconnect Interval set to: 1s
2017/04/17 08:35:17.424970 redis.go:112: INFO [RedisOutput] Using Redis server x.x.x.x:6379
2017/04/17 08:35:17.424974 redis.go:116: INFO [RedisOutput] Redis connection timeout 5s
2017/04/17 08:35:17.424979 redis.go:117: INFO [RedisOutput] Redis reconnect interval 1s
2017/04/17 08:35:17.424983 redis.go:118: INFO [RedisOutput] Using index pattern filebeat-redis
2017/04/17 08:35:17.424987 redis.go:119: INFO [RedisOutput] Topology expires after 15s
2017/04/17 08:35:17.424992 redis.go:120: INFO [RedisOutput] Using db 0 for storing events
2017/04/17 08:35:17.424999 redis.go:121: INFO [RedisOutput] Using db 1 for storing topology
2017/04/17 08:35:17.425006 redis.go:122: INFO [RedisOutput] Using 0 data type


2017/04/17 08:35:22.425215 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s
2017/04/17 08:35:28.425464 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s
2017/04/17 08:35:34.425800 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s
2017/04/17 08:35:40.426412 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s
2017/04/17 08:35:46.426857 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s
2017/04/17 08:35:52.427249 redis.go:178: WARN Error connecting to Redis (dial tcp x.x.x.x:6379: i/o timeout). Retrying in 1s

Thanks
-Raja

[tudor (Tudor Golubenco)]

Please note that, due to exactly this limitation, we plan to remove the option of sending to multiple outputs in Beats 6.0. We currently recommend using Logstash or a queue like Kafka if you need to duplicate the data to multiple outputs. The reason is that in Beats we're trying to keep the outputs as simple as possible, and for any use case that requires this type of functionality we recommend Logstash.