Different index output using ingest pipeline

Elastic Stack Elasticsearch
1

Hello, I'm trying to use ingest pipeline to get different index output.
Here's my case: I have fluentd to hit input on Elasticsearch, fluentd make index input on Elasticsearch named as "app-write", I want to filter from the current source input.
here is glimpse of my index if I dont use any filter:

"_source": {
"kubernetes": {
"container_name": "iris-uat"
} }

I want to get different index output based on value from [kubernetes][container_name]. there is only two value on the current field, it is "iris-uat" and "iris".
Is there any way to get index output as "app-write-iris" and "app-write-iris-uat"?

Keep in mind that I didn't use logstash to filter the index because I'm having much trouble with it (such as overload input or else)

Any help is much appreciated
Thanks

2

You could do something like this - Ingest pipelines | Elasticsearch Guide [7.15] | Elastic

3

I've seen the documentation, but I'm sorry I don't really understand how it will be implemented on my case

"processors": [
    {
      "pipeline": {
        "description": "If 'service.name' is 'apache_httpd', use 'httpd_pipeline'",
        "if": "ctx.service?.name == 'apache_httpd'",
        "name": "httpd_pipeline"
      }
    }

according to the example, can I just change the "name" to index name that I want?

4

The name is the pipeline you want to use. So you'd need 3 pipelines;

  1. The first one which figures out the container_name value
  2. A pipeline for iris that sends to it's own index
  3. A pipeline for iris-uat that sends to it's own index
1 Like
5

from your explanation, I guess this is what I should do

  1. Make pipeline named 'iris-rule' that have conditional to get container name
    (if container name 'iris' go to 'iris' pipeline, and do so for 'iris-uat')
  2. Make pipeline named 'iris' to just ingest input to 'iris' index
  3. Make pipeline named 'iris-uat to just ingest input to 'iris-uat' index

cmiiw, sorry for bothering because I haven't try to make any pipeline before

6

I've created test pipeline based on your instructions, and It works perfectly, but I tried on reroute api, so I rerouted an index to a new index.
so for my case, should I state the pipeline rule's name on my index template to apply the pipeline automatically on document input?

Thanks in advance

7

Ideally, yes.

1 Like
8

thanks for the help, hope you have a nice day

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.