A case for Redis as broker

Elastic Stack Elasticsearch
1

Hi all, we are building an elasticsearch stack currently but hitting some
uncertainty whether to include Redis or not.

I am writing up an argument for it's inclusion and hoped to gain additional
insight from this experienced community.

We are using elasticsearch specifically for indexing various system log
files, and may generate as much as 1gb per hour.

So far, I have made points regarding improved scalability by segmenting
logstash and basically using Redis as a fancy buffer.
Also, I have made the point about flexibility should we ever bring
ElasticSearch down that we continue to capture logs during any downtime.
And the point that we can attach parallel services to either side of Redis.

Are there additional advantage or drawbacks that I should be aware of?

Thanks so much for any insight, thoughts or pointers to relevant existing
conversations on this topic.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I really want to better understand what we gain by storing messages in Redis. Is the relief felt by logstash or by elastic search? How so?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Take a look at this thread
https://groups.google.com/d/topic/elasticsearch/3CaRrizmBcQ/discussion

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 12 November 2013 07:46, Nicholas Rios nicholas.jay.rios@gmail.comwrote:

I really want to better understand what we gain by storing messages in
Redis. Is the relief felt by logstash or by Elasticsearch? How so?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Not sure how much you want to scale....but something like a rabbitmq
cluster seems, to me, far more scalable than redis, for this use case, and
with better data safety rules...
You could of course have multiple redis and hash across the cluster but it
is harder to scale up and down as needed.
Throughput will likely be higher with redis on a simple implementation...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.