Logstash Redis Output - Expire Keys?

titan1978 · March 12, 2019, 6:08pm

We are building a Redis output configuration that pushes a large stack of data to Redis. The volume is so large that we want to set an EXPIRY for each key that is pushed to Redis. This is very important from both costs and maintanence perspective. I could not locate a setting in Logstash (6.4) on how I can do this.

Can someone point on how I can achieve this?

pjanzen · March 12, 2019, 6:56pm

The redis output plugin doesn't support this option and redis doesn't either. So I would say create a feature request.. I don's see any option to set a ttl on a key...

(to be complete)
redis does not support a default TTL on a db key set via a config file. of course it support the TTL on a per key basis..

titan1978 · March 12, 2019, 7:02pm

yep Redis does support on a per-key basis. The scenario we're looking is pretty much like this - We're looking to expire ALL Keys over X days old (can be done on Redis) :

RPUSH somekey 12345 9999
EXPIRE somekey X

pjanzen · March 12, 2019, 7:12pm

yeah I don't know how to do that without scan all the keys and update them with a expire setting. But reading your initial post that will be very costly in resources.

titan1978 · March 12, 2019, 7:26pm

shouldn't it be as simple as if there is a config called 'expiry'

if @config_expiry:
@redis.rpush(key, payload)
@redis.expire(key,exipireVal)
else:
@redis.rpush(key, payload)

pjanzen · March 12, 2019, 7:28pm

yeah I guess so

Christian_Dahlqvist · March 12, 2019, 7:28pm

The Redis plugin writes to either a list or a channel, which as far as I know do not support TTL.

pjanzen · March 12, 2019, 7:32pm

You could always fork the logstash-output-redis on github and extend the code..

github.com

logstash-plugins/logstash-output-redis/blob/master/lib/logstash/outputs/redis.rb#L217


if @batch && @data_type == 'list' # Don't use batched method for pubsub.
  # Stud::Buffer
  buffer_receive(payload, key)
  return
end


begin
  @redis ||= connect
  if @data_type == 'list'
    congestion_check(key)
    @redis.rpush(key, payload)
  else
    @redis.publish(key, payload)
  end
rescue => e
  @logger.warn("Failed to send event to Redis", :event => event,
               :identity => identity, :exception => e,
               :backtrace => e.backtrace)
  sleep @reconnect_interval
  @redis = nil
  retry

titan1978 · March 13, 2019, 12:50pm

yeah that was the exact place i was thinking of doing this. Though It would be preferable to get this as part of an official release? since we'll have to rebuild the plugin each time we upgrade.

system · April 10, 2019, 12:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.