Hi I am using elastic-agent in k8s cluster with kubernetes integration. I have added a custom pipeline for kubernetes container logs to re-route all logs from containers in a specific namespace. The following is the code for my ingest pipeline PUT _ingest/pipeline/logs-kubernetes.container_logs…

[image] stephenb: proxy Thanks Leandro and Stephan. It's working after I upgraded the kubernetes intergration to latest. @stephenb Will try your suggestion of building composable pipelines and update

[image] umesh2020: ctx.kubernetes.namespace =='fi1-https' Namespaces cannot have a - in the name, it is not allowed. You would need to add an extra field and replace the - with a _.

Hi Leandro, Thanks for the help. I updated my ingest pipeline as suggested, but I still see the same issue PUT _ingest/pipeline/logs-kubernetes.container_logs@custom { "processors": [ { "set": { "field": "k8s_elastic_namespace", "value": "{{ kubernetes.namespace }}", …

What do you have in the error.message field? If the reroute processor is failing for any reason, you will have a message in this field because of the global on_failure processor. The reroute processor is pretty simple, there is not much else to configure.

The moment I add reroute processor, all container messages from fi1-https messages are no longer seen. I also don't see the new index created.

You do not have any more messages from your kubernetes after adding the reroute processor? Also, the new index would be a datastream named logs-kubernetes.container_logs-fi1_https. Try to remove all the ignore_failure from the processors in the custom ingest pipeline to see where it fails.

I have messages from other kubernetes namespaces except for fi1-https. Should I look for ignore_failure in logs-* data view or some other data view ?

[image] umesh2020: Should I look for ignore_failure in logs-* data view or some other data view ? Yes, if the reroute processor is failing then you will not have the data stream that you want to create with it, the log will be in the original data stream.

[image] umesh2020: container Looks like I am getting security exception due to permission issues: {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for API key id of user [elastic/fleet-server] on indices [logs-kubernetes.container_logs-fi1-https], th…

Yeah, this is what I was thinking could be the issue. What version of the Stack and the integration are you using? The permissions for your user does not matter, they are not used, the fleet managed Elastic Agent uses API keys for each integration with pretty limited permissions. I'm assuming tha…

[image] umesh2020: The moment I add reroute processor, all container messages from fi1-https messages are no longer seen. I also don't see the new index created. [image] umesh2020: Looks like I am getting security exception due to permission issues: {"type":"security_exception","reason":…

New index not created by ingestion pipeline

Elastic Stack Elasticsearch

leandrojmp (Leandro Pereira) February 3, 2025, 9:56pm 19

Yeah, you need to upgrade to version 1.42.0, after that your reroute processor to change the namespace is expected to work.

Fortigate Integration - Separation of logs datastream based on a field value

Topic		Replies	Views
Sending data to new Data Stream with Elastic Agent Elastic Agent	20	1514	March 26, 2024
Understand fleet pipelines and reroute Elasticsearch	2	327	June 3, 2025
Change destination datastream with Elasticsearch ingest pipeline Elasticsearch docker , ingest-pipeline , datastreams	11	2443	July 28, 2023
Elastic fleet change datastream name based on field value Beats fleet	10	773	May 9, 2022
Elastic Agent, single custom log to multiple indices via ingest pipeline Elasticsearch ingest-pipeline	7	1245	December 22, 2021

New index not created by ingestion pipeline

Related topics