Logstash-Forwarder and Redis

Elastic Stack Logstash
1

I want to try using Redis with logstash-forwarder, but I'm confused on a point.

When forwarder tires to connect it complains about a tls handshake and fails to connect.

Can forwarder connect to redis? This post suggests that for 'forwarder' to Redis the full-on logstash is needed on the client.

As always, thanks in advance.

2

LSF can only talk to LS, it cannot interact with redis.
See here for the docs.

3

Hello Mark,
I am going through the same problem.
I have server where LS is running in shipper and Indexer mode with redis in between.
Shipper--> input= lumberjack and output =redis
Indexer--> input = redis and output = elasticsearch
LSF is running on client machines which talks to LS shipper.

I want LSF to talk with redis so that I can store some log file specific keys and then create those many indexes on the basis of keys (e.g keyName-YYYY-MM-DD)
But As you said LSF cant talk to redis then I have to bring shipper to client side.
So what is benefit of LSF?
How can I create multiple indexes in ES through one LS indexer running on server? sample configuration please?

thanks
Sunil Chaudhari

4

I think LSF's README file explains it's reason for being. See below for how to create different indexing depending on the message.

5

Hi Marko,
Thanks for quick reply.

In the given example, you are creating index on the basis of project name which you take from path.
I want to create indexes from values of 'Application' field which I set in logstash-forwarder on client.
For example:
in LSF config on CRM client 1 I say
{
"paths": [
"/var/log/crm/crmERROR.log"
],
"fields": {"Application":"CRM","Sub-System":"Mysystem", "type":"Error-logs "
}

in LSF config on sales client 2 I say
{
"paths": [
"/var/log/sales/SalesERROR.log"
],
"fields": {"Application":"Sales","Sub-System":"Myststem", "type":"Error-logs "
}

all those logs will be sent to redis through shipper and indexer will pick it up from redis.
Now in LS indexer I want to create indexes on the basis of this field "Application", How Can I get this value of Application field so that I can use it in output to create indexname.

br,
Sunil.

6

Sorry for incorrect name in salutation part.
Please read it as "Magnus"

7

The answer I linked to contains the answer to your question. You can refer to you Application field in exactly the same way.

But let's not hijack the original topic with a completely different question. Please start a new topic if you have follow-up questions.

(Also, note that you can edit your own post if you get something wrong. That's preferably to posting new messages with corrections.)

8

Thank you very much Magnus.
Will use it as directed.