Redis-elasticsearch - Is this recommended? Not sure if it is working

Elastic Stack Elasticsearch
1

Very new to ELK and trying to set up the infrastructure to centrally log our in house application logs. What I am trying to set up looks something like this:

Client Server UI
Filebeat => Logstash => Redis => Elasticsearch <= Kibana < = > httpd
v-1.0.1 v- 2.1.1-1 v-3.0.5 v-2.1.1 v-4.3.1 2.2.15-47

So far I have about 15 app logs making there way to Elasticsearch and are viewable via Kibana. I'm not sure however if Redis is actually in play in my current configuration however. If I cycle all the apps to generate a lot of log activity, then watch the redis queue via redis-cli llen redis, but the queue always remains at zero.

Im guessing I have a configuration error on my part, but was also reading that redis may be deprecated at the current version ? Does this mean it is no longer needed/recommended for as a queuing mechanism for ES?

2

I am not a logstash user, but shouldn't it be
Redis->Logstash->Elasticsearch?

Redis acts like a queue for Logstash inputs. Elasticsearch does not
communicate with Redis, it works with a broker such as Logstash (or custom
code).

3

Thanks for that, as it would explain why redis seems to not be in use :slight_smile:
I think I should have posted in Beats or Logstash, but what I've been able to find online thus far it seems I should just drop Redis from the equation at this point.

4

Yes redis can be an AMQP in front of your logstash indexer(s), in case they get flooded with event data. But filebeat handle retransmission, so redis isn't necessary to avoid loosing events. Even logstash can be left out in case you dont want to tamper even data before storing such as ES documents. As filebeat also can output directly to ES, so simplest would be:

filebeat ( -> ES

alternativt:

filebeat -> logstash (filtering events) -> ES

See the paragraph 'The logs are the queue' here