AWS RDS Postgres Provided Grok expressions do not match field value

I'm trying to setup Filebeat with the postgres module to publish logs from an AWS/RDS instance to elasticsearch. I have a python script which can download the logs and the output looks like this:

2018-04-18 08:10:50 UTC:111.222.333.444(54888):root@postgres:[11162]:LOG:  statement: SELECT

Filebeat is able to publish the data to elasticsearch and when I view it in Kibana I see the following error:

 "error": {
  "message": "Provided Grok expressions do not match field value:

Trying to diagnose the problem my self and I was able to publish a normal postgres log from another server which wasnt and RDS instance.

The output for that log was:
2018-04-19 08:11:37.248 UTC [4369] mhowland@postgres ERROR: permission denied for schema pgagent

I noticed the output were different and had different fields so I created a GROK filter and found this works:

%{DATESTAMP:timestamp} %{TZ}:%{IP:ip_address}%{DATA:thread_id}:%{DATA:user}@%{DATA:database}:%{DATA:connection_id}:%{DATA:level}:  %{GREEDYDATA:message}

But this is where I am stuck on what to do next with this filter. It would be good to know if I am doing this correctly should I be using logstash? The documentation for the postgresql modules says I should be publishing the data straight to elasticsearch.

First post on here so apologies in advance if I have posted this incorrectly

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.