Config Redis - Logstash - ElasticSearch

Elastic Stack Beats
1

Hello,
After testing the Logstash output for PacketBeat, I went to try the Redis output. I think the Redis output might still be the best option for production since it can give a lot of buffering.

I could integrate packets using Packetbeat -> Redis -> LogStash -> ElasticSearch. The problem comes from two output fields of Logstash that do not behave as usual.
Here is the Logstash config file:

input {
  redis {
    host => "127.0.0.1"
    port => 6379
    codec => "json"
    data_type => "list"
    type => "redis-input"
    key => "packetbeat"
  }
}

output {
  elasticsearch {
    protocol => "http"
    host => "localhost"
    port => "9200"
    sniffing => true
    flush_size => 100000
    manage_template => false
    index => "%{[@metadata][index]}"
    document_type => "%{[@metadata][type]}"
  }
}
    ... 
   and then some filters

I would like to prevent Logstash from overwriting the fields _index and _type. Usually it is done with:

output {
  elasticsearch {
    ...
    index => "%{[@metadata][index]}"
    document_type => "%{[@metadata][type]}"
  }
}

But when I put those lines in the config file of Logstash, here is what I get in ElasticSearch for each document:

"_index": "%{[@metadata][index]}",
"_type": "%{[@metadata][type]}",
"_id": "AVClKH-LQPDvKznld5wM",
_version": 1,
"_score": 1,
"_source": { 
...

For some reason "%{[@metadata][index]}" and "%{[@metadata][type]}" were not interpreted in the config file and the expression appears in the values of _index and _type. I get this behaviour when using Redis as input for Logstash.

Btw, I don't know if this topic should be labeled Packetbeat or Logstash.

1 Like
2

[@metadata][index] and [@metadata][type] are only send by the logstash outputer in packetbeat. When sending events via redis these fields are missing => broken '_index' and '_type' fields.

You have similar problems when sending to logstash directly?

3

Thanks for your answer @steffens.

When sending to Logstash everything is fine. But we really need to use Redis for the production environnement.

Why do these fields don't exist for the Redis outputer ?
I saw an update https://github.com/elastic/libbeat/pull/231 that I first thought would help with this issue but this update focus on the Logstash outputer.

It's written in the libbeat Redis outputer that Redis is depreciated and is replaced by "the Logstash Output that has support for Redis Output plugin."

So that means we can only use Redis after Logstash ? With something that look like that:
Packetbeat -> Logstash -> Redis -> ElasticSearch

But we actually need to buffer packets before Logstash to make sure we won't lose any traffic. So the way to go is Packetbeat -> Redis -> Logstash -> ElasticSearch

4

@metadata is special to Logstash, as it is handled special from within logstash. The @metadata is normally dropped itself by Logstash when publishing.

When routing events via redis these fields are missing on purpose as we have no assumptions what will happen with these events. If you need a special pattern for indexing in logstash you can try using using filters in logstash. Optionally packetbeat can publish some tags.

5

Ok, I did that. Result is not super clean but that will do. Thank you for the advice.

So I guess the Redis outputer won't be maintained ? It appears depreciated. It there a reason for that ? We might reconsider using Redis if you think its best to use Logstash. But how can we face sudden traffic peaks ?

6

may I point you to this github discussion on redis support? https://github.com/elastic/filebeat/issues/132

As long as redis output is in codebase it is supposed to work, but you are right it is deprecated and might be remove in the future. But not adding @metadata to redis outputer was on purpose.

1 Like