I'm trying to setup Filebeat with the postgres module to publish logs from an AWS/RDS instance to elasticsearch. I have a python script which can download the logs and the output looks like this:
2018-04-18 08:10:50 UTC:111.222.333.444(54888):root@postgres:[11162]:LOG: statement: SELECT
Filebeat is able to publish the data to elasticsearch and when I view it in Kibana I see the following error:
"error": {
"message": "Provided Grok expressions do not match field value:
Trying to diagnose the problem my self and I was able to publish a normal postgres log from another server which wasnt and RDS instance.
The output for that log was:
2018-04-19 08:11:37.248 UTC [4369] mhowland@postgres ERROR: permission denied for schema pgagent
I noticed the output were different and had different fields so I created a GROK filter and found this works:
%{DATESTAMP:timestamp} %{TZ}:%{IP:ip_address}%{DATA:thread_id}:%{DATA:user}@%{DATA:database}:%{DATA:connection_id}:%{DATA:level}: %{GREEDYDATA:message}
But this is where I am stuck on what to do next with this filter. It would be good to know if I am doing this correctly should I be using logstash? The documentation for the postgresql modules says I should be publishing the data straight to elasticsearch.
First post on here so apologies in advance if I have posted this incorrectly