# New index not created by ingestion pipeline
**URL:** https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018
**Category:** Elasticsearch
**Tags:** ingest-pipeline
**Created:** [February 3, 2025, 3:48pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018 "2025-02-03T15:48:26Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 3:48pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/1 "2025-02-03T15:48:26Z")
Hi
I am using elastic-agent in k8s cluster with kubernetes integration. I have added a custom pipeline for kubernetes container logs to re-route all logs from containers in a specific namespace. The following is the code for my ingest pipeline
```auto
PUT _ingest/pipeline/logs-kubernetes.container_logs@custom
{
"processors": [
{
"reroute": {
"namespace": [
"{{ kubernetes.namespace }}"
],
"if": "((ctx?.kubernetes?.namespace != null) && (ctx.kubernetes.namespace =='fi1-https'))"
}
}
]
}
```
When I test the pipeline, I am able to see that pipeline is expected to create a new index. But after I save the pipeline, I don't see the new index created. Can you shared some tips on how to debug this issue and if there is anything wrong in my pipeline ?
```auto
{
"docs": [
{
"processor_results": [
{
"processor_type": "reroute",
"status": "success",
"if": {
"condition": "((ctx?.kubernetes?.namespace != null) && (ctx.kubernetes.namespace =='fi1-https'))",
"result": true
},
"doc": {
"_index": ".ds-kubernetes.container_logs-fi1-https",
"_version": "1",
"_id": "c8RmzJQBRUQciHblRJBL",
"_source": {
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 4:01pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/2 "2025-02-03T16:01:53Z")
> [@umesh2020](#):
>
> `ctx.kubernetes.namespace =='fi1-https'`
Namespaces cannot have a `-` in the name, it is not allowed.
You would need to add an extra field and replace the `-` with a `_`.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 4:58pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/3 "2025-02-03T16:58:51Z")
Hi Leandro,
Thanks for the help. I updated my ingest pipeline as suggested, but I still see the same issue
```auto
PUT _ingest/pipeline/logs-kubernetes.container_logs@custom
{
"processors": [
{
"set": {
"field": "k8s_elastic_namespace",
"value": "{{ kubernetes.namespace }}",
"if": "((ctx?.kubernetes?.namespace != null) && (ctx.kubernetes.namespace.contains('fi1-https')))",
"ignore_failure": true
}
},
{
"gsub": {
"field": "k8s_elastic_namespace",
"pattern": "-",
"replacement": "_",
"ignore_missing": true,
"if": "ctx.k8s_elastic_namespace != null",
"ignore_failure": true
}
},
{
"reroute": {
"namespace": [
"{{k8s_elastic_namespace}}"
],
"if": "ctx.k8s_elastic_namespace != null",
"ignore_failure": true
}
}
]
}
```
Test pipeline output shows proper index
```auto
{
"docs": [
{
"doc": {
"_index": ".ds-kubernetes.container_logs-fi1_https",
"_version": "1",
"_id": "w4y8zJQB-OigOW8EhVWn",
"_source": {
"container": {
"image": {
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 6:13pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/4 "2025-02-03T18:13:58Z")
What do you have in the `error.message` field?
If the `reroute` processor is failing for any reason, you will have a message in this field because of the global `on_failure` processor.
The `reroute` processor is pretty simple, there is not much else to configure.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 6:58pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/5 "2025-02-03T18:58:38Z")
The moment I add reroute processor, all container messages from fi1-https messages are no longer seen. I also don't see the new index created.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 7:06pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/6 "2025-02-03T19:06:05Z")
You do not have any more messages from your kubernetes after adding the reroute processor?
Also, the new index would be a datastream named `logs-kubernetes.container_logs-fi1_https`.
Try to remove all the `ignore_failure` from the processors in the custom ingest pipeline to see where it fails.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 7:08pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/7 "2025-02-03T19:08:19Z")
I have messages from other kubernetes namespaces except for fi1-https. Should I look for ignore\_failure in logs-\* data view or some other data view ?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 7:10pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/8 "2025-02-03T19:10:07Z")
> [@umesh2020](#):
>
> Should I look for ignore\_failure in logs-\* data view or some other data view ?
Yes, if the `reroute` processor is failing then you will not have the data stream that you want to create with it, the log will be in the original data stream.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 8:29pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/9 "2025-02-03T20:29:04Z")
> [@umesh2020](#):
>
> container
Looks like I am getting security exception due to permission issues:
{"type":"security\_exception","reason":"action [indices:admin/auto\_create] is unauthorized for API key id of user [elastic/fleet-server] on indices [logs-kubernetes.container\_logs-fi1-https], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]"}
I created the ingest pipeline as a superuser, but still I am seeing this error.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 9:11pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/10 "2025-02-03T21:11:58Z")
Yeah, this is what I was thinking could be the issue.
What version of the Stack and the integration are you using?
The permissions for your user does not matter, they are not used, the fleet managed Elastic Agent uses API keys for each integration with pretty limited permissions.
I'm assuming that you are using an old Kubernetes integration version, these permissions are available from version `1.42.0` as you can check it [here](https://github.com/elastic/integrations/blob/537f9df7677e2b0a580880af48cfad4164166a01/packages/kubernetes/changelog.yml#L202-L206).
````auto
- version: "1.42.0"
changes:
- description: Add permissions to reroute events to logs-*-* for container_logs datastream
type: enhancement
link: https://github.com/elastic/integrations/pull/6340```
````
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 3, 2025, 9:18pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/11 "2025-02-03T21:18:22Z")
> [@umesh2020](#):
>
> The moment I add reroute processor, all container messages from fi1-https messages are no longer seen. I also don't see the new index created.
> [@umesh2020](#):
>
> Looks like I am getting security exception due to permission issues:
> {"type":"security\_exception","reason":"action [indices:admin/auto\_create] is unauthorized for API key id of user [elastic/fleet-server] on indices [logs-kubernetes.container\_logs-fi1-https], this action is granted by the index privileges [auto\_configure,create\_index,manage,all]"}
>
> I created the ingest pipeline as a superuser, but still I am seeing this error.
> [@leandrojmp](#):
>
> The permissions for your user does not matter, they are not used, the feet managed Elastic Agent uses API keys for each integration with pretty limited permissions.
You can not do this... sorry Fleet + Agent create very strict API Keys under the covers that does not allow rerouting to other data streams.
>
>
> \## Description
>
> To allow to use a reroute permissions, the api key used to writ…e data need to have the permissions to write to the destination datastreams.
>
> This is currently problematic in Fleet, as we try to limit the needed permissions to the integration policy datastreams, to support this we should allow user to add additional datastreams permissions.
>
>
> \## API Changes
>
> \- \[\] We need to a new property to the package policy API \`additional\_datastreams\_permissions\` that allow to give additional permissions
>
> \`\`\`
> POST kbn:/api/fleet/package\_policies
> {
> ...,
> "additional\_datastreams\_permissions": \["logs-test-default", "metrics-test-\*"\]
>
> }
> \`\`\`
>
> \- \[\] The \`additional\_datastreams\_permissions\` should be validated against the space \`allowed\_namespace\_prefix\` https://github.com/elastic/kibana/pull/188003
>
>
> \- \[\] The function that generate permission for the \`.fleet-policies\` document should be updated to add permissions for those additional datastreams https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/agent\_policies/package\_policies\_to\_agent\_permissions.ts#L55
>
> \- \[\] We should add a e2e test that create a package policy, a custom ingest pipeline with a reroute processor and verify the permissions of the \`.fleet-policies\` document allow the reroute to work as expected
>
> \## UI Changes
>
>
> \- \[\] The package policy editor need to be updated with a combo box that allow to provide a list of additionnal datastream
>
> !\[Image\](https://github.com/user-attachments/assets/3415569d-f572-49f4-9fb3-34841139471e)
>
> \- \[\] The pipeline UI could be updated
> !\[Image\](https://github.com/user-attachments/assets/c8261c92-7951-4972-bbfd-6afc825159ae)
@umesh2020 What are you actually trying to accomplish?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 9:19pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/12 "2025-02-03T21:19:57Z")
> [@stephenb](#):
>
> You can not do this... sorry Fleet + Agent create very strict API Keys under the covers that does not allow rerouting to other data streams.
>
> [[Fleet] Allow to add additional datastream permission to use reroute processor · Issue #203585 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/203585)
Some integrations and data streams already can do that, which is the case of the `kubernetes.container_logs`.
This was added in this [PR](https://github.com/elastic/integrations/pull/6340).
I'm using this on production for other integrations like the Kafka Custom Logs and the Cloudwatch Logs.
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 3, 2025, 9:35pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/13 "2025-02-03T21:35:14Z")
Ahh yes LOL I am using on one of my K8s now that I'm looking... But when I saw his permission error... The Agent API keys are usually what caused that.
If you go outside of the certain bounds, that's when you'll run into trouble.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 9:36pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/14 "2025-02-03T21:36:48Z")
We have a multi-tenant application running in k8s cluster. Each tenant application runs in separate k8s namespace, so I want to store logs specific to each tenant in a separate index. Then I plan to create a separate kibana namespace per tenant and provide access to only logs of that tenant
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 3, 2025, 9:38pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/15 "2025-02-03T21:38:47Z")
Let me check when I get back to my desk...
And technically you're trying to create different data streams...
I think you'll want to use namespace when I get back to my desk I can triple check. See what I did
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 9:42pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/16 "2025-02-03T21:42:36Z")
What is the version of your Kubernetes integration in Fleet?
What you want to do, reroute to a different namespace, is possible, but the integration version needs to be at least `1.42.0`.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 9:45pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/17 "2025-02-03T21:45:48Z")
> [@leandrojmp](#):
>
> Kubernetes
OK. That's what the reroute is supposed to do right ? I am using version 1.36, will upgrade and let you know
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [February 3, 2025, 9:54pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/18 "2025-02-03T21:54:41Z")
So here is how I do it... I like building Composable Pipelines...
```auto
PUT _ingest/pipeline/kubernetes.container_logs@custom
{
"processors": [
{
"set": {
"field": "event.dataset",
"ignore_empty_value": true,
"if": "ctx?.event?.dataset == null",
"copy_from": "data_stream.dataset"
}
},
{
"pipeline": {
"name": "sendtoistio",
"if": "ctx?.kubernetes?.container?.name == 'istio-proxy' || ctx?.k8s?.container?.name == 'istio-proxy'"
}
}
]
}
GET _ingest/pipeline/sendtoistio
{
"processors": [
{
"set": {
"field": "data_stream.dataset",
"value": "istio.access_logs"
}
},
{
"set": {
"field": "data_stream.namespace",
"value": "default"
}
},
{
"set": {
"field": "event.dataset",
"value": "{{data_stream.dataset}}"
}
},
{
"reroute": {
"dataset": "{{data_stream.dataset}}",
"namespace": "{{data_stream.namespace}}"
}
}
]
}
```
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [February 3, 2025, 9:56pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/19 "2025-02-03T21:56:00Z")
> [@umesh2020](#):
>
> I am using version 1.36, will upgrade and let you know
Yeah, you need to upgrade to version 1.42.0, after that your `reroute` processor to change the namespace is expected to work.
---
### Author:  [@umesh2020](https://discuss.elastic.co/u/umesh2020)
#### Post date: [February 3, 2025, 10:24pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018/20 "2025-02-03T22:24:06Z")
> [@stephenb](#):
>
> `proxy`
Thanks Leandro and Stephan. It's working after I upgraded the kubernetes intergration to latest. @stephenb Will try your suggestion of building composable pipelines and update