Logs not being indexed in Elasticsearch from Redis

Elastic Stack Logstash
1

Hi all,

Trying to build the following pipeline for apache access logs: Filebeat -> Redis -> Logstash -> Elasticsearch.

All components are configured (see below configs) and start successfully: Filebeat publishes the logs to an "apache" channel on Redis and Logstash subscribes to the channel. A Logstash index is created on Elasticsearch but it remains empty of documents. Logstash, Elasticsearch and Redis logs contain no errors.

Configurations:

Filebeat:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/apache2/access.log

output.redis:
  hosts: ["localhost"]
  key: "apache"
  db: 0
  timeout: 5

Logstash:

input {
  redis {
    host => "localhost"
    key => "apache"
    data_type => "channel"
    codec => json
  }
}

filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  geoip {
      source => "clientip"
    }
}

output {
  elasticsearch { 
    hosts => ["localhost:9200"] 
  }
}

An example of a messages in the Redis channel:

1562657837.663939 [0 127.0.0.1:33972] "RPUSH" "apache" "{\"@timestamp\":\"2019-07-09T07:37:16.662Z\",\"@metadata\":{\"beat\":\"filebeat\",\"type\":\"_doc\",\"version\":\"7.2.0\"},\"log\":{\"offset\":1984,\"file\":{\"path\":\"/var/log/apache2/access.log\"}},\"message\":\"91.205.154.22 - - [09/Jul/2019:07:37:15 +0000] \\\"GET /hello.html HTTP/1.1\\\" 304 180 \\\"-\\\" \\\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36\\\"\",\"input\":{\"type\":\"log\"},\"agent\":{\"id\":\"736b2ac9-9062-4705-9405-f2233250a82e\",\"version\":\"7.2.0\",\"type\":\"filebeat\",\"ephemeral_id\":\"d000dc8d-83f3-4975-83d0-db7f85c2167e\",\"hostname\":\"ip-172-31-26-146\"},\"ecs\":{\"version\":\"1.0.0\"},\"host\":{\"name\":\"ip-172-31-26-146\"}}"

Thanks!

2

Beats is pushing to the tail of a LIST and, with data_type => "channel", Logstash is subscribing to a CHANNEL - 2 different data structures in Redis. You need data_type => "list".

3

Thank you! Yes, that seems to have been the issue. Added data_type: "list" to the Filebeat config and defined the same value for the same setting in Logstash as well. That works now.

Oddly though, setting the value to channel in both Filebeat and Logstash doesn't seem to work, which would be my preferred method to use Redis Pub/Sub. Any idea why?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.