Logstash Redis Output - Expire Keys?

We are building a Redis output configuration that pushes a large stack of data to Redis. The volume is so large that we want to set an EXPIRY for each key that is pushed to Redis. This is very important from both costs and maintanence perspective. I could not locate a setting in Logstash (6.4) on how I can do this.

Can someone point on how I can achieve this?

The redis output plugin doesn't support this option and redis doesn't either. So I would say create a feature request.. I don's see any option to set a ttl on a key...

(to be complete)
redis does not support a default TTL on a db key set via a config file. of course it support the TTL on a per key basis..

yep Redis does support on a per-key basis. The scenario we're looking is pretty much like this - We're looking to expire ALL Keys over X days old (can be done on Redis) :

RPUSH somekey 12345 9999
EXPIRE somekey X

yeah I don't know how to do that without scan all the keys and update them with a expire setting. But reading your initial post that will be very costly in resources.

shouldn't it be as simple as if there is a config called 'expiry'

if @config_expiry:
@redis.rpush(key, payload)
@redis.rpush(key, payload)

yeah I guess so :slight_smile:

The Redis plugin writes to either a list or a channel, which as far as I know do not support TTL.

You could always fork the logstash-output-redis on github and extend the code..

yeah that was the exact place i was thinking of doing this. Though It would be preferable to get this as part of an official release? since we'll have to rebuild the plugin each time we upgrade.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.