How does Filebeat Redis module work for Slow Logs?

dandago · December 8, 2019, 9:21pm

The configuration file of the Redis module for Filebeat seems to support normal logs (from the Redis server's log file) and slowlogs (from the API)... see:

github.com

elastic/beats/blob/master/filebeat/modules.d/redis.yml.disabled

# Module: redis
# Docs: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-redis.html

- module: redis
  # Main logs
  log:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths: ["/var/log/redis/redis-server.log*"]

  # Slow logs, retrieved via the Redis API (SLOWLOG)
  slowlog:
    enabled: true

    # The Redis hosts to connect to.
    #var.hosts: ["localhost:6379"]

    # Optional, the password to use when connecting to Redis.

This file has been truncated. show original

I can understand how the logs are picked up from the log files, but I had no idea that Filebeat had the capability to issue a command to a server and thus retrieve information. Is there any documentation on how this feature works, and in what format the slowlogs are shipped?

ChrsMark · December 9, 2019, 9:34am

Hi!

You can find the Module's documentation here: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-redis.html

Thanks!

dandago · December 12, 2019, 11:36pm

Hi, thanks for the doc link. I understand where Filebeat is getting the slowlogs from, however I was looking for more detail in how this works in Filebeat, given that Filebeat is meant to monitor files. I was also looking for the format/structure in which they are shipped.

ChrsMark · December 13, 2019, 9:13am

Hey,

Filebeat can also retrieve "logs" from remote systems like hosts or queues like Kafka, or Amazon S3. The idea is that everything that can be parsed as log line can be parsed by Filebeat.

Regarding the fields you can find the list here.

Thanks!

dandago · December 13, 2019, 9:38am

Awesome, thanks!

Is it required to use the Redis filebeat module for shipping slowlogs, or can it be configured without it?

I'm looking to ship Redis logs to a third party managed ELK, so it's not possible to set up the pipeline and resources that the module seems to expect.

ChrsMark · December 13, 2019, 10:04am

Hmm,

you need a way to collect the logs and send them to the Logstash node of ELK right? If so , yeah you need Filebeat on the node you want to monitor.

dandago · December 13, 2019, 10:16am

Yes, send them to Logstash node of ELK, that's right. I can use Filebeat to ship Redis logs, but is it also possible to ship the slowlogs without being able to set up the environment (pipeline etc)?

ChrsMark · December 13, 2019, 1:04pm

You can send the events from Filebeat directly to Elasticsearch if you want to by-pass Logstash. I don't see a way to avoid using Filebeat.

Let me know!

dandago · December 13, 2019, 1:23pm

I'm saying to avoid using the Redis module, not avoid using Filebeat. Filebeat is a must. Logstash is irrelevant.

ChrsMark · December 13, 2019, 1:25pm

Got it! In order to get slowlogs you need Redis Module yeah.

dandago · December 13, 2019, 1:48pm

Great, thanks for clarifying!

system · January 10, 2020, 1:48pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.