Different index output using ingest pipeline

[alfianaf (Alfian Aulia Firdaus)]

Hello, I'm trying to use ingest pipeline to get different index output.
Here's my case: I have fluentd to hit input on Elasticsearch, fluentd make index input on Elasticsearch named as "app-write", I want to filter from the current source input.
here is glimpse of my index if I dont use any filter:

"_source": {
"kubernetes": {
"container_name": "iris-uat"
} }

I want to get different index output based on value from [kubernetes][container_name]. there is only two value on the current field, it is "iris-uat" and "iris".
Is there any way to get index output as "app-write-iris" and "app-write-iris-uat"?

Keep in mind that I didn't use logstash to filter the index because I'm having much trouble with it (such as overload input or else)

Any help is much appreciated
Thanks

[warkolm (Mark Walkom)]

You could do something like this - Ingest pipelines | Elasticsearch Guide [7.15] | Elastic

[alfianaf (Alfian Aulia Firdaus)]

I've seen the documentation, but I'm sorry I don't really understand how it will be implemented on my case

"processors": [
    {
      "pipeline": {
        "description": "If 'service.name' is 'apache_httpd', use 'httpd_pipeline'",
        "if": "ctx.service?.name == 'apache_httpd'",
        "name": "httpd_pipeline"
      }
    }

according to the example, can I just change the "name" to index name that I want?

[warkolm (Mark Walkom)]

The name is the pipeline you want to use. So you'd need 3 pipelines;

1. The first one which figures out the container_name value
2. A pipeline for iris that sends to it's own index
3. A pipeline for iris-uat that sends to it's own index

[alfianaf (Alfian Aulia Firdaus)]

from your explanation, I guess this is what I should do

1. Make pipeline named 'iris-rule' that have conditional to get container name
   (if container name 'iris' go to 'iris' pipeline, and do so for 'iris-uat')
2. Make pipeline named 'iris' to just ingest input to 'iris' index
3. Make pipeline named 'iris-uat to just ingest input to 'iris-uat' index

cmiiw, sorry for bothering because I haven't try to make any pipeline before

[alfianaf (Alfian Aulia Firdaus)]

I've created test pipeline based on your instructions, and It works perfectly, but I tried on reroute api, so I rerouted an index to a new index.
so for my case, should I state the pipeline rule's name on my index template to apply the pipeline automatically on document input?

Thanks in advance

[warkolm (Mark Walkom)]

@alfianaf

so for my case, should I state the pipeline rule's name on my index template to apply the pipeline automatically on document input?

Ideally, yes.

[alfianaf (Alfian Aulia Firdaus)]

thanks for the help, hope you have a nice day