Very strange ingest pipeline problem

Elastic Stack Elasticsearch
1

I have a need to add an ingest pipeline to the elastic-cloud-logs-9 data stream in a cloud-hosted environment running 9.2. It’s not complicated logic, but it’s a little complicated extracting a field from message with 2 different formats. I couldn’t get this to work, even though the simulate works, the template was updated and the index rolled over.

Then it gets stranger.

I added a SET to add a tag. just to show the pipeline was entered. Some of the events are tagged, the ones I want aren’t. Thinking my script was an error, I removed that (script) processor. So I have a pipeline with no logic, just adds a tag. Still some are tagged, some aren’t.

It looks only event.dataset == agent.log events go thru the pipeline, events that I want are from elasticsearch.server. There are others that aren’t tagged as well, elasticsearch.gc, elasticsearch.audit and kibana.log. I think these agent.log events come from one host, the other events are from other hosts, I think ALL are the cloud hosts.

I have a case open with Elastic, but I just wondered if anyone here has any ideas.

Thanks

2

Can you post the pipeline and some data that reproduces the problem?

3

I didn’t post here to try to “end around” support, I’m just stumped on how this is even possible at this point.

Thanks

The pipeline

"authentication-failure-pipeline": {
"description": "Pipeline to process authentication failure messages",
"on_failure": [
{
"set": {
"field": "pipeline_error",
"value": "Error in ingest pipeline authentication-failure-pipeline"
}
}
],
"created_date_millis": 1765140238638,
"processors": [
{
"set": {
"field": "tags",
"value": "pipeline-entered"
}
}
],
"modified_date_millis": 1765286917103
}

Events (With a little sanitation)

Tagged:

{
"_index": ".ds-elastic-cloud-logs-9-2025.12.09-000016",
"_id": "DCrCCJsBC3OazxdRX2lq",
"_version": 1,
"_source": {
"agent": {
"name": "4e67b71c189f",
"id": "7ce2d76b-b30c-475e-a861-dcf5ae9a63eb",
"type": "filebeat",
"ephemeral_id": "287746bd-3654-48b9-9b1b-be8ac989d827",
"version": "9.2.0"
},
"service.name": "fleet-server",
"log": {
"file": {
"inode": "30933294",
"path": "/app/elastic-agent/data/logs/elastic-agent-20251210-29.ndjson",
"device_id": "64769",
"fingerprint": "978c4ada193def59063011f0bb516a3226cc5f5022b10aa6d5348a92cea46533"
},
"offset": 9256711,
"source": {
"address": "fleet-server-es-containerhost"
}
},
"http.request.id": "01KC4C4PPQF94NM7R02PBEDXFG",
"fleet.access.apikey.id": "yyyyyy",
"message": "applying new components data",
"tags": "pipeline-entered",
"server.address": "172.17.0.28:8220",
"cloud": {
"availability_zone": "us-east-2a"
},
"service.type": "fleet-server",
"input": {
"type": "filestream"
},
"component": {
"binary": "fleet-server",
"id": "fleet-server-es-containerhost",
"type": "fleet-server",
"dataset": "elastic_agent.fleet_server"
},
"@timestamp": "2025-12-10T14:55:13.801Z",
"ecs": {
"version": "8.0.0"
},
"ecs.version": "1.6.0",
"service": {
"node": {
"name": "instance-0000000023"
},
"name": "xxxx",
"id": "7…f5",
"type": "agent",
"version": "9.2.0"
},
"host": {
"name": "4e67b71c189f"
},
"log.level": "info",
"event": {
"dataset": "agent.log"
},
"fleet.agent.id": "d82b84a0-9f58-41fe-8699-3d277820012c"
},
"fields": {
"component.binary": [
"fleet-server"
],
"service.id": [
"70b88c884d8c4b1db451045acbc590f5"
],
"http.request.id": [
"01KC4C4PPQF94NM7R02PBEDXFG"
],
"service.node.name": [
"instance-0000000023"
],
"cloud.availability_zone": [
"us-east-2a"
],
"server.address": [
"172.17.0.28:8220"
],
"service.type": [
"agent",
"fleet-server"
],
"agent.type": [
"filebeat"
],
"component.id": [
"fleet-server-es-containerhost"
],
"component.dataset": [
"elastic_agent.fleet_server"
],
"log.file.device_id": [
"64769"
],
"log.level": [
"info"
],
"agent.name": [
"4e67b71c189f"
],
"host.name": [
"4e67b71c189f"
],
"fleet.agent.id": [
"d82b84a0-9f58-41fe-8699-3d277820012c"
],
"service.name": [
"xxx",
"fleet-server"
],
"input.type": [
"filestream"
],
"fleet.access.apikey.id": [
"yyyyyyyyyy"
],
"log.offset": [
9256711
],
"message": [
"applying new components data"
],
"tags": [
"pipeline-entered"
],
"component.type": [
"fleet-server"
],
"@timestamp": [
"2025-12-10T14:55:13.801Z"
],
"agent.id": [
"7….eb"
],
"service.version": [
"9.2.0"
],
"ecs.version": [
"8.0.0",
"1.6.0"
],
"log.file.inode": [
"30933294"
],
"log.source.address": [
"fleet-server-es-containerhost"
],
"log.file.path": [
"/app/elastic-agent/data/logs/elastic-agent-20251210-29.ndjson"
],
"agent.ephemeral_id": [
"287746bd-3654-48b9-9b1b-be8ac989d827"
],
"agent.version": [
"9.2.0"
],
"log.file.fingerprint": [
"978c4ada193def59063011f0bb516a3226cc5f5022b10aa6d5348a92cea46533"
],
"event.dataset": [
"agent.log"
]
}
}

Not tagged:

{
"_index": ".ds-elastic-cloud-logs-9-2025.12.09-000016",
"_id": "GbrDCJsBHgGBUBO9Umzo",
"_version": 1,
"_source": {
"agent": {
"name": "1992bf2c59f1",
"id": "5766c01c-062e-4e5a-a895-bf6a044465fa",
"ephemeral_id": "fcedda88-0a53-4c6b-970c-5eefc26a5d34",
"type": "filebeat",
"version": "9.2.0"
},
"process": {
"pid": "246"
},
"log": {
"file": {
"path": "/app/logs/gc.log"
},
"offset": 4738248,
"level": "info"
},
"fileset": {
"name": "gc"
},
"message": "Safepoint \"G1PauseCleanup\", Time since last: 105366142 ns, Reaching safepoint: 20108 ns, At safepoint: 127758 ns, Leaving safepoint: 8919 ns, Total: 156785 ns, Threads: 0 runnable, 100 total",
"cloud": {
"availability_zone": "us-east-2b"
},
"input": {
"type": "log"
},
"@timestamp": "2025-12-10T14:56:16.147Z",
"ecs": {
"version": "1.12.0"
},
"elasticsearch": {
"gc": {
"tags": [
"safepoint"
]
}
},
"service": {
"node": {
"name": "instance-0000000012"
},
"id": "70b88c884d8c4b1db451045acbc590f5",
"type": "elasticsearch",
"version": "9.2.0"
},
"host": {
"name": "1992bf2c59f1"
},
"event": {
"ingested": "2025-12-10T14:56:17.128270346Z",
"created": "2025-12-10T14:56:17.117Z",
"kind": "metric",
"module": "elasticsearch",
"category": "database",
"type": "info",
"dataset": "elasticsearch.gc"
},
"deployment": {
"name": "xxxx"
}
},
"fields": {
"event.category": [
"database"
],
"service.id": [
"70b88c884d8c4b1db451045acbc590f5"
],
"service.node.name": [
"instance-0000000012"
],
"process.pid": [
246
],
"cloud.availability_zone": [
"us-east-2b"
],
"service.type": [
"elasticsearch"
],
"agent.type": [
"filebeat"
],
"event.module": [
"elasticsearch"
],
"log.level": [
"info"
],
"agent.name": [
"1992bf2c59f1"
],
"host.name": [
"1992bf2c59f1"
],
"event.kind": [
"metric"
],
"fileset.name": [
"gc"
],
"input.type": [
"log"
],
"elasticsearch.gc.tags": [
"safepoint"
],
"log.offset": [
4738248
],
"message": [
"Safepoint \"G1PauseCleanup\", Time since last: 105366142 ns, Reaching safepoint: 20108 ns, At safepoint: 127758 ns, Leaving safepoint: 8919 ns, Total: 156785 ns, Threads: 0 runnable, 100 total"
],
"event.ingested": [
"2025-12-10T14:56:17.128Z"
],
"@timestamp": [
"2025-12-10T14:56:16.147Z"
],
"agent.id": [
"5….65fa"
],
"service.version": [
"9.2.0"
],
"ecs.version": [
"1.12.0"
],
"event.created": [
"2025-12-10T14:56:17.117Z"
],
"event.type": [
"info"
],
"log.file.path": [
"/app/logs/gc.log"
],
"agent.ephemeral_id": [
"…….a5d34"
],
"agent.version": [
"9.2.0"
],
"deployment.name": [
"xxxx"
],
"event.dataset": [
"elasticsearch.gc"
]
}
}

4

Where did you configure this pipeline to be executed? In which setting?

5

In a component template:

"index": {
"default_pipeline": "authentication-failure-pipeline"

Which I attached to the the elastic-cloud-logs-9 template, then force a rollover of the data stream.

GET index/_settings shows:

…

    "number_of_shards": "1",
    "max_docvalue_fields_search": "200",
    "default_pipeline": "authentication-failure-pipeline"

That’s part of what puzzles me, if I didn’t have the index settings right to invoke the pipeline, none of the events would be tagged. I’m pretty sure no other process sets a tag value “pipeline-entered” that I just made up, but just to make sure, I checked and that value never exists before I started working on this.

Could the other events be specifying a pipeline at ingest? So the default doesn’t get invoked?

6

OK, I figured it out. I didn’t do the setup on this cloud stack. I looked at the Elasticsearch integration and it wasn’t in use. So I didn’t think any ingest pipelines were in play. I’ve since found that what looks like the “old style” filebeat module is running on the cloud servers and the data goes thru ingest pipelines named like filebeat-9.2.0.elasticsearch-server-pipeline.

When those pipelines are explicitly called by filebeat at ingest, the default pipeline specified in the index doesn’t run. I should have used index.final_pipeline instead.

I’d never noticed that option, I’ve never NEEDED that option before.

Thanks for the help.

1 Like