# Override pipeline on modules not working

**URL:** <https://discuss.elastic.co/t/override-pipeline-on-modules-not-working/382233>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 25, 2025, 4:35pm UTC](https://discuss.elastic.co/t/override-pipeline-on-modules-not-working/382233 "2025-09-25T16:35:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![R\_V](https://avatars.discourse-cdn.com/v4/letter/r/7993a0/32.png) [@R\_V](https://discuss.elastic.co/u/R_V)\
**Post date:** [September 25, 2025, 4:35pm UTC](https://discuss.elastic.co/t/override-pipeline-on-modules-not-working/382233/1 "2025-09-25T16:35:42Z")

</div>

Hello

I've got the following flow

```none
GCP PubSub => Filebeat (9.1.3) => Elasticsearch

```

In the filebeat source, i'm sending postgresql module to GCP pubsub.  
The flow is working fine i'm ingesting logs in elasticsearch, but I wanted to parse better the logs. Currently the message contains all information and it do not populate other fields like `postgresql.log.timestamp`, or `postgresql.log.database`

I created a simple pipeline and added it in the filebeat configuration to ensure it's working first

```json
// GET _ingest/pipeline/filebeat-postgresql-standard-log-pipeline
{
  "filebeat-postgresql-standard-log-pipeline": {
    "description": "Pipeline for parsing PostgreSQL logs using the following log_line_prefix parameter: %t [%p]: [%l-1] user=%u,db=%d,client=%h",
    "on_failure": [
      {
        "set": {
          "field": "error.message",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ],
    "processors": [
      {
        "set": {
          "if": "ctx?.service?.type == 'postgresql'",
          "field": "postgresql.parsing",
          "value": "working"
        }
      }
    ]
  }
}

```

Here is my filebeat 9.1.3 configuration

```yaml
filebeat.inputs:
  - type: gcp-pubsub
    project_id: "<project_id>"
    topic: "<topic_name>"
    subscription.name: "<subscription_name>"
    subscription.create: false
    credentials_file: "/usr/share/filebeat/gcp-pubsub-credentials.json"
    processors:
      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true
          add_error_key: true
          expand_keys: true

setup.ilm.enabled: false
setup.template.enabled: false

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  pipelines:
    - pipeline: "filebeat-postgresql-standard-log-pipeline"
      when.contains:
        service.type: "postgresql"
  indices:
    - index: "filebeat-%{[agent.version]}"
      when.contains:
        agent.type: "filebeat"

```

I've simulated the ingestion in `POST _ingest/pipeline/filebeat-postgresql-standard-log-pipeline/_simulate` and the pipeline is working fine.  
But when I ingest real logs, it does not work.

Here the log I'm ingesting in the pubsub topic

```json
{
  "@timestamp": "2025-09-25T14:03:22.296Z",
  "event": {
    "module": "postgresql",
    "dataset": "postgresql.log"
  },
  "input": {
    "type": "log"
  },
  "host": {
    "name": "localhost"
  },
  "tags": [
    "beats_input_codec_plain_applied"
  ],
  "fileset": {
    "name": "log"
  },
  "message": "2025-09-25 14:03:17 UTC [2175]: [1-1] user=,db=,client= FATAL: archive command failed with exit code 127",
  "@version": "1",
  "agent": {
    "ephemeral_id": "b4fccc1d-1bad-4298-840c-7af097b13195",
    "type": "filebeat",
    "hostname": "localhost",
    "id": "46eb61dc-b764-4bdd-92b9-6d8316da100e",
    "name": "localhost",
    "version": "7.10.2"
  },
  "service": {
    "type": "postgresql"
  },
  "log": {
    "offset": 209370,
    "file": {
      "path": "/var/log/postgresql/postgresql-15-main.log"
    }
  },
  "ecs": {
    "version": "1.5.0"
  }
}

```

I've read this post ([Ingest pipeline not working for filebeat](https://discuss.elastic.co/t/ingest-pipeline-not-working-for-filebeat/228015)) saying I can add it my index but this is not the behavior I need as I'll have multiple pipeline based on the module (system, postgresql, nginx, etc.)

We found a working workaround by adding a processor in the gcp-pubsub input processors but we don't know if this is the right thing to do.

```yaml
- add_fields:
    target: "@metadata"
    fields:
      pipeline: filebeat-postgresql-standard-log-pipeline
    when:
      equals:
        event.module: postgresql

```

Did I miss something in my configuration ?

Thanks in advance

---

<div class="post-metadata">

**Author:** ![Musab\_Dogan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/musab_dogan/32/70691_2.png) [@Musab\_Dogan](https://discuss.elastic.co/u/Musab_Dogan)\
**Post date:** [September 29, 2025, 9:48am UTC](https://discuss.elastic.co/t/override-pipeline-on-modules-not-working/382233/2 "2025-09-29T09:48:14Z")

</div>

Can you try to add `dot.expander` processor ?

```auto
PUT _ingest/pipeline/filebeat-postgresql-standard-log-pipeline
{
  "description": "Pipeline for parsing PostgreSQL logs using the following log_line_prefix parameter: %t [%p]: [%l-1] user=%u,db=%d,client=%h",
  "on_failure": [
    {
      "set": {
        "field": "error.message",
        "value": "{{ _ingest.on_failure_message }}"
      }
    }
  ],
  "processors": [
    {
      "dot_expander": {
        "field": "postgresql.parsing"
      }
    },
    {
      "set": {
        "if": "ctx?.service?.type == 'postgresql'",
        "field": "postgresql.parsing",
        "value": "working"
      }
    }
  ]
}

```

> **[Dot expander processor | Reference](https://www.elastic.co/docs/reference/enrich-processor/dot-expand-processor)**
>
> Expands a field with dots into an object field. This processor allows fields with dots in the name to be accessible by other processors in the pipeline...