Logstash rds filtering?

[jspriesterbach (James Spriesterbach)]

Hello,

I currently have a script that will download all the error & slow logs from RDS for the day and put them in a folder. I am now trying to get logstash to read those files and put them into our ELK stack for use with kibana. Does anyone have an example filter that would work for RDS/mysql logs? at the moment I was just trying a simple:

filter {
if [type] == "rds_logs" {
grok {
match => { "message" => "%{RDS_LOGS}"}
}
date {
locale => "en"
match => ["timestamp", "d/MMM/YYYY:HH:mm:ss Z"]
}
}
}

But of course %RDS_LOGS is not defined because there seems to be no grok filters built into logstash for rds. Heres an example log:

$ cat mysql-error-running.log.2018-07-29.01
2018-07-29 00:45:38 2b7ebf3cf400 InnoDB: Warning: Setting innodb_checksums to OFF is DEPRECATED. This option may be removed in future releases. You should set innodb_checksum_algorithm=NONE instead.
2018-07-29 00:45:38 8513 [Note] InnoDB: Started in read only mode
2018-07-29 00:45:38 8513 [Note] InnoDB: The InnoDB memory heap is disabled
2018-07-29 00:45:38 8513 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-07-29 00:45:38 8513 [Note] InnoDB: Compressed tables use zlib 1.2.3
2018-07-29 00:45:38 8513 [Note] InnoDB: CPU does not support crc32 instructions

Thanks

[warkolm (Mark Walkom)]

A quick web search returned https://www.phase2technology.com/blog/adding-mysql-slow-query-logs-logstash

[Badger]

For the error logs something like this should get you started

    grok { 
        match => { 
            "message" => [ 
                 "%{DATE:date} %{TIME:time} %{NUMBER:randomNumber} \[%{WORD:loglevel}\] InnoDB: %{GREEDYDATA:logtext}",
                 "%{DATE:date} %{TIME:time} %{BASE16NUM:randomHexString} InnoDB: %{GREEDYDATA:logtext}"
            ]
        }
    }