When using the ingest-pipeline that filebeat creates for postgresql logs, the resulting event contains the field postgresql.log.timestamp which conflicts with the ecs@mappings component template.
The filebeat-9.5.4-postgresql-log pipeline includes the following "date" processor:
{
"date": {
"formats": [
"yyyy-MM-dd HH:mm:ss.SSS zz",
"yyyy-MM-dd HH:mm:ss zz"
],
"field": "postgresql.log.timestamp",
"target_field": "@timestamp"
}
}
Which sets the @timestamp field from postgresql.log.timestamp, but the field itself is passed along without any modification, which causes an indexing failure because of the ecs_date template in ecs@mappings.
Is it intentional that the ingest pipeline does not either drop the postgresql.log.timestamp field, or also converts that to a valid date type?
Right now I've had to add an additional custom pipeline that drops the field, (since the original data is already available as @timestamp, but I would have expected that the pipeline supplied by filebeat would produce something that worked with ECS out of the box.