# Filebeat PostgreSQL module not parsing message

**URL:** <https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [March 17, 2021, 9:52am UTC](https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464 "2021-03-17T09:52:54Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![martinfrancois](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinfrancois/32/85635_2.png) [@martinfrancois](https://discuss.elastic.co/u/martinfrancois)\
**Post date:** [March 17, 2021, 9:52am UTC](https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464/1 "2021-03-17T09:52:55Z")

</div>

We are using filebeat 7.11.0 along with the `postgresql` module. However for some reason, the `message` attribute seems not to be touched, although some metadata seems to be added by the module.

Our log messages look to be the same format as the one used in the [tests](https://github.com/elastic/beats/blob/bb789311a28422b2b0a3487bf51006faa4025856/filebeat/module/postgresql/log/test/postgresql-11.4.log) but don't result in the same [expected output](https://github.com/elastic/beats/blob/bb789311a28422b2b0a3487bf51006faa4025856/filebeat/module/postgresql/log/test/postgresql-11.4.log-expected.json).

The logs get sent from filebeat to our logstash, where we see the same log message arrive as the one sent by filebeat in the filebeat debug log.

In the filebeat logs, it seems like it finds the `postgresql.yml` configuration file and also picks up new logs added to the postgresql file, but the event seems to be published without a parsed `message`.

For example, this is the line in the log file (names of tables etc. masked):

```auto
2021-03-16 13:35:12.231 UTC [37] LOG: statement: SELECT "x"."y", "x"."z", "x"."expire_date" FROM "x" WHERE ("x"."expire_date" > '2021-03-16T13:35:12.190372+00:00'::timestamptz AND "x"."y" = 'xyz') LIMIT 21

```

And this is the result if I turn debug logging on in filebeat:

```auto
2021-03-16T15:02:31.269Z DEBUG [processors] processing/processors.go:203 Publish event: {
  "@timestamp": "2021-03-16T15:02:31.268Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.11.0",
    "pipeline": "filebeat-7.11.0-postgresql-log-pipeline"
  },
  "event": {
    "module": "postgresql",
    "dataset": "postgresql.log"
  },
  "fileset": {
    "name": "log"
  },
  "deployment": "staging",
  "message": "2021-03-16 13:35:12.231 UTC [37] LOG: statement: SELECT \"x\".\"y\", \"x\".\"z\", \"x\".\"expire_date\" FROM \"x\" WHERE (\"x\".\"expire_date\" > '2021-03-16T13:35:12.19
0372+00:00'::timestamptz AND \"x\".\"y\" = 'xyz') LIMIT 21",
  "service": {
    "type": "postgresql"
  }
}

```

> **We are using filebeat dockerized, the entry in docker-compose.yml looks like this:**
>
> ```auto
> filebeat:
> restart: unless-stopped
> image: docker.elastic.co/beats/filebeat:7.11.0
> user: root
> environment:
> - DEPLOYMENT_NAME
> volumes:
> - db_logs:/var/log/postgresql:ro
> - ./services/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
> - ./services/filebeat/modules.d:/usr/share/filebeat/modules.d:ro
> - /var/lib/docker/containers:/var/lib/docker/containers:ro
> - /var/run/docker.sock:/var/run/docker.sock:ro
> command: >
> filebeat -strict.perms=false
> -E output.logstash.hosts=["${LOGSTASH_HOST}"]
> -E output.logstash.ssl.verification_mode=full
> 
> ```

> **Our filebeat.yml:**
>
> ```auto
> filebeat.config:
> modules:
> path: ${path.config}/modules.d/*.yml
> reload.enabled: false
> 
> filebeat.autodiscover:
> providers:
> - type: docker
> hints.enabled: true
> hints.default_config.enabled: false
> 
> processors:
> - drop_fields:
> fields:
> - agent
> - docker
> - ecs
> - host
> - input
> - log
> - stream
> - timestamp
> ignore_missing: true
> - add_fields:
> target: ''
> fields:
> deployment: ${DEPLOYMENT_NAME:n/a}
> 
> logging.metrics.enabled: false
> 
> monitoring.enabled: false
> 
> ```

Just to make sure it's not related to our `processors`, I tried removing them, but this lead to the same result.

> **Our modules.d/postgresql.yml:**
>
> ```auto
> - module: postgresql
> log:
> enabled: true
> var.paths: ["/var/log/postgresql/postgresql-*.log"]
> 
> ```

Since the full filebeat log is too long for this post, I uploaded it here: [2021-03-16T15:02:30.839Z INFO instance/beat.go:660 Home path: [/usr - Pastebin.com](https://pastebin.com/bK8B9k47)

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [March 17, 2021, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464/2 "2021-03-17T17:43:01Z")

</div>

Hi @martinfrancois, welcome to the Elastic community forums!

The `postgresql` Filebeat module parses the logs using [Elasticsearch ingest pipelines](https://github.com/elastic/beats/tree/master/filebeat/module/postgresql/log/ingest). No parsing is done in Filebeat itself. This is why you are seeing the `message` field as-is and no additional parsed fields coming out of Filebeat or in Logstash.

You have two ways to solve this parsing problem:

- either send these logs directly from Filebeat to Elasticsearch, in which case the appropriate ingest pipelines will be automatically loaded into Elasticsearch for you and parsing will happen before documents are indexed.
- or keep sending the logs from Filebeat to Logstash, but then use the `elasticsearch` output in Logstash to send the logs to Elasticsearch. Also, make sure to [manually load](https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html) the necessary ingest pipelines.

Hope this helps,

Shaunak

---

<div class="post-metadata">

**Author:** ![martinfrancois](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martinfrancois/32/85635_2.png) [@martinfrancois](https://discuss.elastic.co/u/martinfrancois)\
**Post date:** [March 19, 2021, 9:01am UTC](https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464/3 "2021-03-19T09:01:37Z")

</div>

Hi @shaunak thanks for the welcome!

Thanks for your answer, using your instruction I was able to set it up and get it working properly!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 16, 2021, 11:01am UTC](https://discuss.elastic.co/t/filebeat-postgresql-module-not-parsing-message/267464/4 "2021-04-16T11:01:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.