# PostgreSQL module got error when parsing AWS RDS PostgreSQL log **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 17, 2018, 2:07pm UTC](https://discuss.elastic.co/t/postgresql-module-got-error-when-parsing-aws-rds-postgresql-log/136209 "2018-06-17T14:07:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![nyamada2](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@nyamada2](https://discuss.elastic.co/u/nyamada2)\ **Post date:** [June 17, 2018, 2:07pm UTC](https://discuss.elastic.co/t/postgresql-module-got-error-when-parsing-aws-rds-postgresql-log/136209/1 "2018-06-17T14:07:40Z")
I wrote a script that downloads AWS RDS PostgreSQL log files and let the filebeat reads log files using PostgreSQL module, though I found it fails to parse in kibana dashboard. There was a similar post before at below URL link. > [@AWS RDS Postgres Provided Grok expressions do not match field value](https://discuss.elastic.co/t/aws-rds-postgres-provided-grok-expressions-do-not-match-field-value/129116): > > I'm trying to setup Filebeat with the postgres module to publish logs from an AWS/RDS instance to elasticsearch. I have a python script which can download the logs and the output looks like this: 2018-04-18 08:10:50 UTC:111.222.333.444(54888):root@postgres:[11162]:LOG: statement: SELECT Filebeat is able to publish the data to elasticsearch and when I view it in Kibana I see the following error: "error": { "message": "Provided Grok expressions do not match field value: Trying to diagnose … AWS RDS PostgreSQL does not allow the users to change log\_line\_prefix parameter, so it should conform this parameter. `%t:%r:%u@%d:[%p]:` When I use PostgreSQL module with default settings, I found that json messages at "[filebeat PostgreSQL] Overview" in the Dashboard page. ```auto { "_index": "filebeat-6.3.0-2018.06.17", "_type": "doc", "_id": "8RDYDWQBmFa0p5c-iVek", "_version": 1, "_score": null, "_source": { "offset": 738, "prospector": { "type": "log" }, "source": "/var/log/rds_postgres/test/2018-06-14/postgresql.log.2018-06-14-19", "message": "2018-06-14 19:11:26 UTC::@:[3707]:LOG: checkpoint starting: time", "fileset": { "module": "postgresql", "name": "log" }, "error": { "message": "Provided Grok expressions do not match field value: [2018-06-14 19:11:26 UTC::@:[3707]:LOG: checkpoint starting: time]" }, "input": { "type": "log" }, "@timestamp": "2018-06-17T13:04:11.151Z", "beat": { "hostname": "ip-172-31-24-211", "name": "ip-172-31-24-211", "version": "6.3.0" }, "host": { "name": "ip-172-31-24-211" } }, "fields": { "@timestamp": [ "2018-06-17T13:04:11.151Z" ] }, "highlight": { "fileset.module": [ "@kibana-highlighted-field@postgresql@/kibana-highlighted-field@" ] }, "sort": [ 1529240651151 ] } ``` As this message says, it fails. Then, I think grok pattern should be like ```auto "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone}:%{IPORHOST:postgresql.log.clientip}:%{USERNAME:postgresql.log.user}@%{HOSTNAME:postgresql.log.database}:\\[%{INT:postgresql.log.process_id}\\]:%{WORD:postgresql.log.level}: %{GREEDYDATA:log_message}", "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone}:%{IPORHOST:postgresql.log.clientip}:@:\\[%{INT:postgresql.log.process_id}\\]:%{WORD:postgresql.log.level}: %{GREEDYDATA:log_message}", "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone}:%{IPORHOST:postgresql.log.clientip}:%{USERNAME:postgresql.log.user}@%{HOSTNAME:postgresql.log.database}:\\[%{INT:postgresql.log.process_id}\\]:%{WORD:postgresql.log.level}: duration: %{NUMBER:postgresql.log.duration} ms statement: %{MULTILINEQUERY:postgresql.log.query}", "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone}:%{IPORHOST:postgresql.log.clientip}:@:\\[%{INT:postgresql.log.process_id}\\]:%{WORD:postgresql.log.level}: duration: %{NUMBER:postgresql.log.duration} ms statement: %{MULTILINEQUERY:postgresql.log.query}" ``` But I don't know where I can override grok pattern settings of PostgreSQL module. Any one can help me? ---
**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\ **Post date:** [June 18, 2018, 3:48am UTC](https://discuss.elastic.co/t/postgresql-module-got-error-when-parsing-aws-rds-postgresql-log/136209/2 "2018-06-18T03:48:44Z")
Hi, You can create a new pipeline with your requirement and define the same in filebeat.yml file with pipeline tag as below. ``` # The Ingest Node pipeline ID associated with this prospector. If this is set, it # overwrites the pipeline option from the Elasticsearch output. pipeline: ``` Please check below link for more understanding. [https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html](https://www.elastic.co/guide/en/beats/filebeat/master/configuring-ingest-node.html) Regards, ---
**Author:** ![nyamada2](https://avatars.discourse-cdn.com/v4/letter/n/d07c76/32.png) [@nyamada2](https://discuss.elastic.co/u/nyamada2)\ **Post date:** [June 20, 2018, 2:42pm UTC](https://discuss.elastic.co/t/postgresql-module-got-error-when-parsing-aws-rds-postgresql-log/136209/3 "2018-06-20T14:42:55Z")
Thanks! I'll carefully read that document and implement it! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 18, 2018, 4:43pm UTC](https://discuss.elastic.co/t/postgresql-module-got-error-when-parsing-aws-rds-postgresql-log/136209/4 "2018-07-18T16:43:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.