# Moving from single redis instances to clustered caching pool **URL:** \ **Category:** Logstash\ **Created:** [August 19, 2015, 9:29pm UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713 "2015-08-19T21:29:51Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [August 19, 2015, 9:29pm UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/1 "2015-08-19T21:29:51Z")
We currently run single instance of redis to cache logs before sending to Elasticsearch. We want to move to a cluster to avoid single point of failure. At this point it feels like we need to move away from Redis due to the following reasons 1) the cluster implementation is very new and probably not mature 2) Logstash plugins for redis are not cluster away from what i see in the github trackers. So the question is what is the best queue cluster for logstash? There are many articles out there but they dont talk about this use case specifically. I read that kafka can handle a ton of messages per second but it doesnt guarantee the order of messages is maintained (which is very important in logs). I also read that rabbitmq does not handle as many messages etc etc. Basically the goal is to go from 1 redis server to 4-5 product X servers. Messages would be clustered so losing nodes would not take down the stack. Logstash would be able to resiliently pull from any of the servers. ---
**Author:** ![Jakauppila](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakauppila/32/44935_2.png) [@Jakauppila](https://discuss.elastic.co/u/Jakauppila)\ **Post date:** [August 20, 2015, 12:55am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/2 "2015-08-20T00:55:36Z")
How many messages a second are you expecting? Assuming the events are timestamped, exact order of messages shouldn't be an issue since they'll be indexed based on that value. ---
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [August 20, 2015, 1:35am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/3 "2015-08-20T01:35:29Z")
according to marvel my elasticsearch is getting up to 75000 messages per second. unfortunately not all logs have a full timestamp. It only has hour,min secs with no date. Without the date it is hard to parse the timestamp. Cannot assume current day either. ---
**Author:** ![widhalm\_t](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/widhalm_t/32/4290_2.png) [@widhalm\_t](https://discuss.elastic.co/u/widhalm_t)\ **Post date:** [August 20, 2015, 11:18am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/4 "2015-08-20T11:18:49Z")
I would not use Redis clustered but instead use some Redis nodes and have my Logstash `redis` outputs failover between them. You can use one `redis` output in your Logstash configuration and Logstash will pick one and send all event to it. If Redis fails, Logstash will pick another one out of the list and send to this one. Don't configure more `redis` outputs to avoid duplicate messages, but use one output with multiple Redis hosts in the `host` directive. In a normal setup, the messages are read and deleted from Redis before the cluster could even synchronize them to other nodes. ---
**Author:** ![allenmchan](https://avatars.discourse-cdn.com/v4/letter/a/ec9cab/32.png) [@allenmchan](https://discuss.elastic.co/u/allenmchan)\ **Post date:** [August 20, 2015, 4:41pm UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/5 "2015-08-20T16:41:19Z")
I didnt realize this is the behavior of the redis output as the docs didnt explicitly mention that. The redis input host field is only a string so it can only pull from one redis server. That means i need to double my logstash indexers while half of them are idle? Unless there are no other viable solution ( from the list of rabbitmq, kafka etc) then i guess this will be the route i have to go to keep HA ---
**Author:** ![widhalm\_t](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/widhalm_t/32/4290_2.png) [@widhalm\_t](https://discuss.elastic.co/u/widhalm_t)\ **Post date:** [August 21, 2015, 7:36am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/6 "2015-08-21T07:36:24Z")
No, you can add multiple `redis` inputs to your Indexers. You have one `redis` output with multiple hosts in the `host` directive and multiple `redis` input with one host each. So every Indexer will try to pull from every Redis in parallel. This way you can add and remove Indexers without changing anything in the configuration of your other services. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:31am UTC](https://discuss.elastic.co/t/moving-from-single-redis-instances-to-clustered-caching-pool/27713/7 "2017-07-06T05:31:23Z")