Logstash + Redis

Elastic Stack Logstash
1

Hi guys,
I am trying to figure out how to use Logstash with Redis
Does anyone know how to configure logstash to insert a specific field in message in Redis ?

For example, I have a json message with some fields, like: Name, Passport Number, Age, ..
I would like to insert in Redis (using redis output plugin) only the Passport Number (as Key) and Name (as Value)

2

Simply use the mutate filter remove processor and remove the fields you do not want to send to redis

3

A prune filter with a whitelist_names option is another alternative.

4

@stephenb @Badger
I cant do it.
There is other ouput where i need to send all the information.
If i use mutate filter, all outputs will be affected.

5

Use pipeline-to-pipeline communication with a forked-path pattern.

6

Nice, i will take a look in your suggestion.

7

I think that you can also use the codec option in the redis output to format the output message.

Something like this:

codec => plain { format => "%{field1} %{field2} %{fieldN}" }

But if you have different outputs, I would say that the pipeline-to-pipeline communication is the best approach.

8

Just to update, the solution i found to my case was put the following lines in the filter:

  • to insert data in Redis (inserting data from diameter_Session-Id and msisdn fields):
                if "3868" in [tcp_srcport] {
                    ruby {
                        init => 'require "redis"; $rc = Redis.new(host: "127.0.0.1", port: 6379)'
                        code => '
                            $rc.sadd(event.get("diameter_Session-Id"), event.get("msisdn"))
                            $rc.expire(event.get("diameter_Session-Id"), 3600)
                        '
                    }
                }
  • to get data in Redis (retrieving data from Redis using the value of diameter_Session-Id field as key):
                if "3868" in [tcp_dstport] {
                    ruby {
                        init => 'require "redis"; $rc = Redis.new(host: "127.0.0.1", port: 6379)'
                        code => 'event.set("msisdn_ruby", $rc.smembers(event.get("diameter_Session-Id")))'
                    }
                }
9

A @@class variable should work. I don't think you ever need to use a $global variable.