Logstash rds filtering?


I currently have a script that will download all the error & slow logs from RDS for the day and put them in a folder. I am now trying to get logstash to read those files and put them into our ELK stack for use with kibana. Does anyone have an example filter that would work for RDS/mysql logs? at the moment I was just trying a simple:

filter {
if [type] == "rds_logs" {
grok {
match => { "message" => "%{RDS_LOGS}"}
date {
locale => "en"
match => ["timestamp", "d/MMM/YYYY:HH:mm:ss Z"]

But of course %RDS_LOGS is not defined because there seems to be no grok filters built into logstash for rds. Heres an example log:

$ cat mysql-error-running.log.2018-07-29.01
2018-07-29 00:45:38 2b7ebf3cf400 InnoDB: Warning: Setting innodb_checksums to OFF is DEPRECATED. This option may be removed in future releases. You should set innodb_checksum_algorithm=NONE instead.
2018-07-29 00:45:38 8513 [Note] InnoDB: Started in read only mode
2018-07-29 00:45:38 8513 [Note] InnoDB: The InnoDB memory heap is disabled
2018-07-29 00:45:38 8513 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-07-29 00:45:38 8513 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-07-29 00:45:38 8513 [Note] InnoDB: CPU does not support crc32 instructions


A quick web search returned https://www.phase2technology.com/blog/adding-mysql-slow-query-logs-logstash


For the error logs something like this should get you started

    grok { 
        match => { 
            "message" => [ 
                 "%{DATE:date} %{TIME:time} %{NUMBER:randomNumber} \[%{WORD:loglevel}\] InnoDB: %{GREEDYDATA:logtext}",
                 "%{DATE:date} %{TIME:time} %{BASE16NUM:randomHexString} InnoDB: %{GREEDYDATA:logtext}"
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.