ERR Failed to RPUSH to redis list (read tcp 10.50.12.200:48343->172.27.30.102:6379: i/o timeout) with %!v(MISSING)

Elastic Stack Beats
1

Hi,

I'm running filebeat version 5.0.0-beta1 on my local machine, with Redis output. Redis is running on an EC2 instance in AWS. I get the following errors in the filebeat log:

ERR Failed to RPUSH to redis list (read tcp 10.50.12.200:48343->172.27.30.102:6379: i/o timeout) with %!v(MISSING)

Redis is accessible from my local machine.

I've tied metricbeat as well and the result is the same.

Increasing the filebeat timeout to redis doesn't seems to make big difference.

The error comes from time to time and otherwise data seems to be flowing correctly as we can see in the log below:

2016-10-10T14:18:41+02:00 INFO Non-zero metrics in the last 30s: libbeat.redis.publish.write_bytes=235481 publish.events=646 registrar.writes=6 libbeat.publisher.published_events=321 libbeat.redis.publish.read_bytes=30 registrar.states.update=646
2016-10-10T14:19:11+02:00 INFO Non-zero metrics in the last 30s: libbeat.redis.publish.write_bytes=264985 libbeat.publisher.published_events=373 registrar.writes=6 publish.events=678 libbeat.redis.publish.read_bytes=31 registrar.states.update=678
2016-10-10T14:19:41+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=5 libbeat.redis.publish.read_bytes=25 publish.events=641 registrar.states.update=641 libbeat.publisher.published_events=391 libbeat.redis.publish.write_bytes=274907
2016-10-10T14:19:58+02:00 ERR Failed to RPUSH to redis list (read tcp 10.50.12.200:48343->172.27.30.102:6379: i/o timeout) with %!v(MISSING)
2016-10-10T14:19:58+02:00 INFO Error publishing events (retrying): read tcp 10.50.12.200:48343->172.27.30.102:6379: i/o timeout
2016-10-10T14:20:11+02:00 INFO Non-zero metrics in the last 30s: libbeat.redis.publish.read_bytes=2239 libbeat.redis.publish.read_errors=1 libbeat.publisher.published_events=385 registrar.states.update=838 registrar.writes=5 publish.events=838 libbeat.redis.publish.write_bytes=326368
2016-10-10T14:20:41+02:00 INFO Non-zero metrics in the last 30s: libbeat.redis.publish.read_bytes=32 registrar.writes=6 libbeat.redis.publish.write_bytes=339406 publish.events=838 libbeat.publisher.published_events=472 registrar.states.update=838
2016-10-10T14:21:05+02:00 ERR Failed to RPUSH to redis list (read tcp 10.50.12.200:48566->172.27.30.102:6379: i/o timeout) with %!v(MISSING)
2016-10-10T14:21:05+02:00 INFO Error publishing events (retrying): read tcp 10.50.12.200:48566->172.27.30.102:6379: i/o timeout
2016-10-10T14:21:11+02:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=463 publish.events=387 registrar.states.update=387 registrar.writes=3 libbeat.redis.publish.read_errors=1 libbeat.redis.publish.write_bytes=347150 libbeat.redis.publish.read_bytes=2230
2016-10-10T14:21:41+02:00 INFO Non-zero metrics in the last 30s: publish.events=999 libbeat.redis.publish.write_bytes=401683 registrar.writes=6 libbeat.redis.publish.read_bytes=32 registrar.states.update=999 libbeat.publisher.published_events=457
2016-10-10T14:22:11+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=6 registrar.states.update=666 publish.events=666 libbeat.publisher.published_events=416 libbeat.redis.publish.write_bytes=297372 libbeat.redis.publish.read_bytes=31
2016-10-10T14:22:41+02:00 INFO Non-zero metrics in the last 30s: registrar.states.update=939 libbeat.publisher.published_events=562 registrar.writes=6 libbeat.redis.publish.read_bytes=32 publish.events=939 libbeat.redis.publish.write_bytes=412528
2016-10-10T14:23:11+02:00 INFO Non-zero metrics in the last 30s: registrar.writes=6 libbeat.redis.publish.read_bytes=32 registrar.states.update=881 libbeat.redis.publish.write_bytes=392111 publish.events=881 libbeat.publisher.published_events=551
2016-10-10T14:23:41+02:00 INFO Non-zero metrics in the last 30s: registrar.states.update=469 publish.events=469 libbeat.redis.publish.read_bytes=30 libbeat.redis.publish.write_bytes=156020
2

I'm trying to reproduce this but couldn't so far. Could you share your full config? Also do you have some more info on the redis server? Which version? Config options? Is redis under heavy load? Did you restart redis?

3

I'm using Redis 3.2.3 64 bit running on m4.xlarge instance with 4 2.4GHz CPUs, 16 GB RAM and 100GB SSD, networking is "high" whatever that means according to Amazon.
Note that Redis is running in docker.
There is nothing else coming to Redis, besides the beats from the 2-3 servers with which I'm testing.

Redis config (using defaults for almost everything): http://pastebin.com/wuhjWa8E
FIlebeat config (defaults for almost everything): http://pastebin.com/6D8U16cr
Metricbeat config (defaults for almost everything): http://pastebin.com/fuBneP9M

Tried to restart redis multiple times and also played with filebeat timeouts a lot.

I've also tried to send filebeats from another EC2 instance to the Redis EC2 instance and there wasn't any timeouts at all. Could this mean that the connections get throttled at my end or at Amazons end? If this is the case then this would be strange as I'm sending small amounts of data with not that many connections.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.