# Very strange ingest pipeline problem

**URL:** <https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942>\
**Category:** Elasticsearch\
**Created:** [December 9, 2025, 7:48pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942 "2025-12-09T19:48:03Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [December 9, 2025, 7:48pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/1 "2025-12-09T19:48:03Z")

</div>

I have a need to add an ingest pipeline to the elastic-cloud-logs-9 data stream in a cloud-hosted environment running 9.2. It’s not complicated logic, but it’s a little complicated extracting a field from message with 2 different formats. I couldn’t get this to work, even though the simulate works, the template was updated and the index rolled over.

Then it gets stranger.

I added a SET to add a tag. just to show the pipeline was entered. Some of the events are tagged, the ones I want aren’t. Thinking my script was an error, I removed that (script) processor. So I have a pipeline with no logic, just adds a tag. Still some are tagged, some aren’t.

It looks only event.dataset == agent.log events go thru the pipeline, events that I want are from elasticsearch.server. There are others that aren’t tagged as well, elasticsearch.gc, elasticsearch.audit and kibana.log. I think these agent.log events come from one host, the other events are from other hosts, I think ALL are the cloud hosts.

I have a case open with Elastic, but I just wondered if anyone here has any ideas.

Thanks

---

<div class="post-metadata">

**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\
**Post date:** [December 10, 2025, 2:16pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/2 "2025-12-10T14:16:23Z")

</div>

Can you post the pipeline and some data that reproduces the problem?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [December 10, 2025, 3:03pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/3 "2025-12-10T15:03:15Z")

</div>

I didn’t post here to try to “end around” support, I’m just stumped on how this is even possible at this point.

Thanks

The pipeline

> "authentication-failure-pipeline": {  
> "description": "Pipeline to process authentication failure messages",  
> "on\_failure": [  
> {  
> "set": {  
> "field": "pipeline\_error",  
> "value": "Error in ingest pipeline authentication-failure-pipeline"  
> }  
> }  
> ],  
> "created\_date\_millis": 1765140238638,  
> "processors": [  
> {  
> "set": {  
> "field": "tags",  
> "value": "pipeline-entered"  
> }  
> }  
> ],  
> "modified\_date\_millis": 1765286917103  
> }

Events (With a little sanitation)

Tagged:

> {  
> "\_index": ".ds-elastic-cloud-logs-9-2025.12.09-000016",  
> "\_id": "DCrCCJsBC3OazxdRX2lq",  
> "\_version": 1,  
> "\_source": {  
> "agent": {  
> "name": "4e67b71c189f",  
> "id": "7ce2d76b-b30c-475e-a861-dcf5ae9a63eb",  
> "type": "filebeat",  
> "ephemeral\_id": "287746bd-3654-48b9-9b1b-be8ac989d827",  
> "version": "9.2.0"  
> },  
> "service.name": "fleet-server",  
> "log": {  
> "file": {  
> "inode": "30933294",  
> "path": "/app/elastic-agent/data/logs/elastic-agent-20251210-29.ndjson",  
> "device\_id": "64769",  
> "fingerprint": "978c4ada193def59063011f0bb516a3226cc5f5022b10aa6d5348a92cea46533"  
> },  
> "offset": 9256711,  
> "source": {  
> "address": "fleet-server-es-containerhost"  
> }  
> },  
> "http.request.id": "01KC4C4PPQF94NM7R02PBEDXFG",  
> "fleet.access.apikey.id": "yyyyyy",  
> "message": "applying new components data",  
> "tags": "pipeline-entered",  
> "server.address": "172.17.0.28:8220",  
> "cloud": {  
> "availability\_zone": "us-east-2a"  
> },  
> "service.type": "fleet-server",  
> "input": {  
> "type": "filestream"  
> },  
> "component": {  
> "binary": "fleet-server",  
> "id": "fleet-server-es-containerhost",  
> "type": "fleet-server",  
> "dataset": "elastic\_agent.fleet\_server"  
> },  
> "@timestamp": "2025-12-10T14:55:13.801Z",  
> "ecs": {  
> "version": "8.0.0"  
> },  
> "ecs.version": "1.6.0",  
> "service": {  
> "node": {  
> "name": "instance-0000000023"  
> },  
> "name": "xxxx",  
> "id": "7…f5",  
> "type": "agent",  
> "version": "9.2.0"  
> },  
> "host": {  
> "name": "4e67b71c189f"  
> },  
> "log.level": "info",  
> "event": {  
> "dataset": "agent.log"  
> },  
> "fleet.agent.id": "d82b84a0-9f58-41fe-8699-3d277820012c"  
> },  
> "fields": {  
> "component.binary": [  
> "fleet-server"  
> ],  
> "service.id": [  
> "70b88c884d8c4b1db451045acbc590f5"  
> ],  
> "http.request.id": [  
> "01KC4C4PPQF94NM7R02PBEDXFG"  
> ],  
> "service.node.name": [  
> "instance-0000000023"  
> ],  
> "cloud.availability\_zone": [  
> "us-east-2a"  
> ],  
> "server.address": [  
> "172.17.0.28:8220"  
> ],  
> "service.type": [  
> "agent",  
> "fleet-server"  
> ],  
> "agent.type": [  
> "filebeat"  
> ],  
> "component.id": [  
> "fleet-server-es-containerhost"  
> ],  
> "component.dataset": [  
> "elastic\_agent.fleet\_server"  
> ],  
> "log.file.device\_id": [  
> "64769"  
> ],  
> "log.level": [  
> "info"  
> ],  
> "agent.name": [  
> "4e67b71c189f"  
> ],  
> "host.name": [  
> "4e67b71c189f"  
> ],  
> "fleet.agent.id": [  
> "d82b84a0-9f58-41fe-8699-3d277820012c"  
> ],  
> "service.name": [  
> "xxx",  
> "fleet-server"  
> ],  
> "input.type": [  
> "filestream"  
> ],  
> "fleet.access.apikey.id": [  
> "yyyyyyyyyy"  
> ],  
> "log.offset": [  
> 9256711  
> ],  
> "message": [  
> "applying new components data"  
> ],  
> "tags": [  
> "pipeline-entered"  
> ],  
> "component.type": [  
> "fleet-server"  
> ],  
> "@timestamp": [  
> "2025-12-10T14:55:13.801Z"  
> ],  
> "agent.id": [  
> "7….eb"  
> ],  
> "service.version": [  
> "9.2.0"  
> ],  
> "ecs.version": [  
> "8.0.0",  
> "1.6.0"  
> ],  
> "log.file.inode": [  
> "30933294"  
> ],  
> "log.source.address": [  
> "fleet-server-es-containerhost"  
> ],  
> "log.file.path": [  
> "/app/elastic-agent/data/logs/elastic-agent-20251210-29.ndjson"  
> ],  
> "agent.ephemeral\_id": [  
> "287746bd-3654-48b9-9b1b-be8ac989d827"  
> ],  
> "agent.version": [  
> "9.2.0"  
> ],  
> "log.file.fingerprint": [  
> "978c4ada193def59063011f0bb516a3226cc5f5022b10aa6d5348a92cea46533"  
> ],  
> "event.dataset": [  
> "agent.log"  
> ]  
> }  
> }

Not tagged:

> {  
> "\_index": ".ds-elastic-cloud-logs-9-2025.12.09-000016",  
> "\_id": "GbrDCJsBHgGBUBO9Umzo",  
> "\_version": 1,  
> "\_source": {  
> "agent": {  
> "name": "1992bf2c59f1",  
> "id": "5766c01c-062e-4e5a-a895-bf6a044465fa",  
> "ephemeral\_id": "fcedda88-0a53-4c6b-970c-5eefc26a5d34",  
> "type": "filebeat",  
> "version": "9.2.0"  
> },  
> "process": {  
> "pid": "246"  
> },  
> "log": {  
> "file": {  
> "path": "/app/logs/gc.log"  
> },  
> "offset": 4738248,  
> "level": "info"  
> },  
> "fileset": {  
> "name": "gc"  
> },  
> "message": "Safepoint \"G1PauseCleanup\", Time since last: 105366142 ns, Reaching safepoint: 20108 ns, At safepoint: 127758 ns, Leaving safepoint: 8919 ns, Total: 156785 ns, Threads: 0 runnable, 100 total",  
> "cloud": {  
> "availability\_zone": "us-east-2b"  
> },  
> "input": {  
> "type": "log"  
> },  
> "@timestamp": "2025-12-10T14:56:16.147Z",  
> "ecs": {  
> "version": "1.12.0"  
> },  
> "elasticsearch": {  
> "gc": {  
> "tags": [  
> "safepoint"  
> ]  
> }  
> },  
> "service": {  
> "node": {  
> "name": "instance-0000000012"  
> },  
> "id": "70b88c884d8c4b1db451045acbc590f5",  
> "type": "elasticsearch",  
> "version": "9.2.0"  
> },  
> "host": {  
> "name": "1992bf2c59f1"  
> },  
> "event": {  
> "ingested": "2025-12-10T14:56:17.128270346Z",  
> "created": "2025-12-10T14:56:17.117Z",  
> "kind": "metric",  
> "module": "elasticsearch",  
> "category": "database",  
> "type": "info",  
> "dataset": "elasticsearch.gc"  
> },  
> "deployment": {  
> "name": "xxxx"  
> }  
> },  
> "fields": {  
> "event.category": [  
> "database"  
> ],  
> "service.id": [  
> "70b88c884d8c4b1db451045acbc590f5"  
> ],  
> "service.node.name": [  
> "instance-0000000012"  
> ],  
> "process.pid": [  
> 246  
> ],  
> "cloud.availability\_zone": [  
> "us-east-2b"  
> ],  
> "service.type": [  
> "elasticsearch"  
> ],  
> "agent.type": [  
> "filebeat"  
> ],  
> "event.module": [  
> "elasticsearch"  
> ],  
> "log.level": [  
> "info"  
> ],  
> "agent.name": [  
> "1992bf2c59f1"  
> ],  
> "host.name": [  
> "1992bf2c59f1"  
> ],  
> "event.kind": [  
> "metric"  
> ],  
> "fileset.name": [  
> "gc"  
> ],  
> "input.type": [  
> "log"  
> ],  
> "elasticsearch.gc.tags": [  
> "safepoint"  
> ],  
> "log.offset": [  
> 4738248  
> ],  
> "message": [  
> "Safepoint \"G1PauseCleanup\", Time since last: 105366142 ns, Reaching safepoint: 20108 ns, At safepoint: 127758 ns, Leaving safepoint: 8919 ns, Total: 156785 ns, Threads: 0 runnable, 100 total"  
> ],  
> "event.ingested": [  
> "2025-12-10T14:56:17.128Z"  
> ],  
> "@timestamp": [  
> "2025-12-10T14:56:16.147Z"  
> ],  
> "agent.id": [  
> "5….65fa"  
> ],  
> "service.version": [  
> "9.2.0"  
> ],  
> "ecs.version": [  
> "1.12.0"  
> ],  
> "event.created": [  
> "2025-12-10T14:56:17.117Z"  
> ],  
> "event.type": [  
> "info"  
> ],  
> "log.file.path": [  
> "/app/logs/gc.log"  
> ],  
> "agent.ephemeral\_id": [  
> "…….a5d34"  
> ],  
> "agent.version": [  
> "9.2.0"  
> ],  
> "deployment.name": [  
> "xxxx"  
> ],  
> "event.dataset": [  
> "elasticsearch.gc"  
> ]  
> }  
> }

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 10, 2025, 3:14pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/4 "2025-12-10T15:14:16Z")

</div>

Where did you configure this pipeline to be executed? In which setting?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [December 10, 2025, 3:24pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/5 "2025-12-10T15:24:29Z")

</div>

In a component template:

> "index": {  
> "default\_pipeline": "authentication-failure-pipeline"

Which I attached to the the elastic-cloud-logs-9 template, then force a rollover of the data stream.

GET index/\_settings shows:

> …
> 
> ```auto
> "number_of_shards": "1",
> "max_docvalue_fields_search": "200",
> "default_pipeline": "authentication-failure-pipeline"
> 
> ```

That’s part of what puzzles me, if I didn’t have the index settings right to invoke the pipeline, none of the events would be tagged. I’m pretty sure no other process sets a tag value “pipeline-entered” that I just made up, but just to make sure, I checked and that value never exists before I started working on this.

Could the other events be specifying a pipeline at ingest? So the default doesn’t get invoked?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [December 10, 2025, 5:52pm UTC](https://discuss.elastic.co/t/very-strange-ingest-pipeline-problem/383942/6 "2025-12-10T17:52:31Z")

</div>

OK, I figured it out. I didn’t do the setup on this cloud stack. I looked at the Elasticsearch integration and it wasn’t in use. So I didn’t think any ingest pipelines were in play. I’ve since found that what looks like the “old style” filebeat module is running on the cloud servers and the data goes thru ingest pipelines named like filebeat-9.2.0.elasticsearch-server-pipeline.

When those pipelines are explicitly called by filebeat at ingest, the default pipeline specified in the index doesn’t run. I should have used index.final\_pipeline instead.

I’d never noticed that option, I’ve never NEEDED that option before.

Thanks for the help.