Filebeat postgresql log module produces timestamp fields that are not indexable when using ECS

Elastic Stack Beats
1

When using the ingest-pipeline that filebeat creates for postgresql logs, the resulting event contains the field postgresql.log.timestamp which conflicts with the ecs@mappings component template.

The filebeat-9.5.4-postgresql-log pipeline includes the following "date" processor:

{
    "date": {
      "formats": [
        "yyyy-MM-dd HH:mm:ss.SSS zz",
        "yyyy-MM-dd HH:mm:ss zz"
      ],
      "field": "postgresql.log.timestamp",
      "target_field": "@timestamp"
    }
  }

Which sets the @timestamp field from postgresql.log.timestamp, but the field itself is passed along without any modification, which causes an indexing failure because of the ecs_date template in ecs@mappings.

Is it intentional that the ingest pipeline does not either drop the postgresql.log.timestamp field, or also converts that to a valid date type?

Right now I've had to add an additional custom pipeline that drops the field, (since the original data is already available as @timestamp, but I would have expected that the pipeline supplied by filebeat would produce something that worked with ECS out of the box.

2

Can you share the exactly error you are receiving? It is not clear what is happening.

If there are indexing files on a native module this suggests that there is some issue in the parsing, but you need to share the full error you got.

3

There's not an error as such, but when I look in "Stack management" -> "Data Set Quality", then all documents in the datastream that filebeat is in ingesting to, are marked as degraded because the postgresql.log.timestamp field is ignored as show here:

image
image478×478 24.3 KB

The ingest pipeline works fine. It's just that it does not touch the postgresql.log.timestamp (apart from copying a parsed value to @timestamptimestamptimestamptimestamp) and therefore does not prevent the field postgresql.log.timestamp from being indexed.

This is an example of the log events where the field was ignored:

{
"@timestamp": "2026-09-24T10:43:30.044+02:00",
"agent.ephemeral_id": "6d815540-f0a3-48cc-bc03-e47a08535486",
"agent.id": "cbb0a8e3-f51f-46f1-9213-7411f0ae606b",
"agent.name": "logstash-test",
"agent.name.text": "logstash-test",
"agent.type": "filebeat",
"agent.version": "9.5.4",
"data_stream.type": "logs",
"ecs.version": "1.12.0",
"event.category": "database",
"event.dataset": "postgresql.log",
"event.ingested": "2026-09-24T10:43:35.251+02:00",
"event.kind": "event",
"event.module": "postgresql",
"event.timezone": "UTC",
"event.type": "info",
"fileset.name": "log",
"fileset.name.text": "log",
"host.name": "logstash-test",
"input.type": "log",
"log.file.path": "/var/log/postgresql/postgresql-18-main.log",
"log.file.path.text": "/var/log/postgresql/postgresql-18-main.log",
"log.level": "LOG",
"log.offset": 172470,
"message": "database system is ready to accept connections",
"postgresql.log.database": null,
"process.pid": 485686,
"related.user": null,
"service.type": "postgresql",
"user.name": null,
"user.name.text": null,
"_ignored": "postgresql.log.timestamp"
}

The original log line was this:

2026-09-24 08:43:30.044 UTC [485686] LOG: database system is ready to accept connections

4

And please disregard that "host.name" is set to "logsstash-test". It does not mean that filebeat is shipping logs to logstash. There's no logstash involved in this setup. Filebeat is shipping logs directly to Elasticsearch.