# How to send ingest pipeline output to a index alias name
**URL:** https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697
**Category:** Elasticsearch
**Tags:** ingest-pipeline
**Created:** [May 21, 2021, 8:08pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697 "2021-05-21T20:08:52Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 21, 2021, 8:08pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/1 "2021-05-21T20:08:52Z")
Hi ,
I am trying to send ingest pipeline output to a index alias name , so that the index rollovers correctly.
what is the required processor or how do we do this ?
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 22, 2021, 12:14am UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/2 "2021-05-22T00:14:38Z")
Hi @Kannappan_Somu Welcome to the community.
I think perhaps we need a little clarification more information before we can help.
What components you are using? Filebeat? Logstash? What version are you on?
How did you set up ILM? How did you setup the write alias?
The way in general these work is you configure the component (beats or logstash) to write to the write alias and also configure the ingest pipeline ... the ingest pipeline does not affect the name of index the data is being written to as it is processed _before_ the document is actually written to the index.
Tell us a bit more, share your config / information and perhaps we can help otherwise we don't have enough information to ep
Please format your code with the `` button
---
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 22, 2021, 3:27pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/3 "2021-05-22T15:27:07Z")
Hi , @stephenb
we are using filebeat 7.12.1 . below is the filebeat output configuration which sends to ingest pipeline "test" . Note that in filebeat config , i m not changing any index name so it uses default index name **filebeat-7.12.1** \*
```
output.elasticsearch:
hosts: ["xxxx"]
protocol: "https"
username: "ksenthil"
password: " ******"
pipeline: "test"
ssl.verification_mode: none
```
I have already created the index template and boot strap index like below
```
PUT _template/ksenthil
{
"index_patterns": ["ksenthil*"],
"settings": {
"index" : {
"lifecycle" : {
"name" : "5_mins_retention_prod",
"rollover_alias" : "ksenthil"
},
"routing" : {
"allocation" : {
"require" : {
"data" : "hot"
}
}
}
}
},
"mappings" : { },
"aliases" : { }
}
```
* * *
```
PUT ksenthil-000001
{
"aliases": {
"ksenthil": {
"is_write_index": true
}
}
}
```
Elastic search version is 7.10 , and the test ingest pipeline has a set processor to change the \_index metadata to "ksenthil" (index alias name) , but this makes ingest pipeline to write output in default filebeat-7.12.1 index instead of writing in index "ksenthil-000001" with alias "ksenthil"
is there a way we can rename the index in ingest pipeline and send to index with alias name as reference (like we do in logstash), so that rollover happens automatically
---
### Author:  [@alexsunny123](https://discuss.elastic.co/u/alexsunny123)
#### Post date: [May 22, 2021, 4:16pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/4 "2021-05-22T16:16:04Z")
> [@Kannappan\_Somu](#):
>
> Hi stephen,
>
> we are using filebeat 7.12.1 . below is the filebeat output configuration which sends to ingest pipeline "test" . Note that in filebeat config , i m not changing any index name so it uses default index name **filebeat-7.12.1** \*
>
> ```auto
> output.elasticsearch:
> hosts: ["xxxx"]
> protocol: "https"
> username: "ksenthil"
> password: " ******"
> pipeline: "test"
> ssl.verification_mode: none
>
> ```
>
> I have already created the index template and boot strap index like below
> [https://ometv.onl](https://ometv.onl) [https://chatroulette.top](https://chatroulette.top) [https://omegle.wtf](https://omegle.wtf) [https://bazoocam.cam](https://bazoocam.cam)
>
> ```auto
> PUT _template/ksenthil
> {
> "index_patterns": ["ksenthil*"],
> "settings": {
> "index" : {
> "lifecycle" : {
> "name" : "5_mins_retention_prod",
> "rollover_alias" : "ksenthil"
> },
> "routing" : {
> "allocation" : {
> "require" : {
> "data" : "hot"
> }
> }
> }
> }
> },
> "mappings" : { },
> "aliases" : { }
> }
>
> ```
>
> ```auto
> PUT ksenthil-000001
> {
> "aliases": {
> "ksenthil": {
> "is_write_index": true
> }
> }
> }
>
> ```
>
> Elastic search version is 7.10 , and the test ingest pipeline has a set processor to change the \_index metadata to "ksenthil" (index alias name) , but this makes ingest pipeline to write output in default filebeat-7.12.1 index instead of writing in index "ksenthil-000001" with alias "ksenthil"
>
> is there a way we can rename the index in ingest pipeline and send to index with alias name as reference (like we do in logstash), so that rollover happens automatically
thanks my issue has been fixed.
---
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 22, 2021, 4:51pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/5 "2021-05-22T16:51:08Z")
No its not fixed @alexsunny123 , can you suggest the workaround for above
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 22, 2021, 5:29pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/6 "2021-05-22T17:29:51Z")
Hi @Kannappan_Somu
Here is probably the more correct way to do it. I would not suggest doing it the way your are trying. if you are going to write to different indices with filebeat go all in and set it up properly.
```
# Sample Pipeline
PUT _ingest/pipeline/test-discuss2
{
"description": "Test Pipeline",
"processors": [
{
"set": {
"field": "event.data",
"value": "stephen"
}
}
]
}
# Sample template
PUT _template/stephen
{
"index_patterns": [
"stephen-*"
],
"settings": {
"index": {
"lifecycle": {
"name": "stephen",
"rollover_alias": "stephen"
}
}
},
"mappings": {},
"aliases": {}
}
# Sample Policy
PUT _ilm/policy/stephen
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "5gb",
"max_age": "1d"
}
}
}
}
}
}
# Output section filebeat.yml
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["localhost:9200"]
pipeline: "test-discuss2"
index : "stephen"
setup.ilm.enabled: true
setup.ilm.rollover_alias: "stephen"
setup.ilm.pattern: "000001"
setup.template.name : "stephen"
setup.template.pattern: "stephen"
```
Note if you do this you will not even need to setup up the boot strap index it will do it for your.
I ran
`./filebeat -e`
It setup the bootstrap index and everything ingested the data
```
GET _cat/indices/s*?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open stephen-000001 oAMkIXECQM6qoLnnf0T1zQ 1 1 10 0 40.4kb 40.4kb
GET _cat/aliases/stephen?v
alias index filter routing.index routing.search is_write_index
stephen stephen-000001 - - - true
```
And I see the pipeline ran
```
GET stephen-000001/_search
....
},
"event" : {
"data" : "stephen"
}
}
....
```
And the setting look right
```
GET stephen-000001
{
"stephen-000001" : {
"aliases" : {
"stephen" : {
"is_write_index" : true
}
},
"mappings" :
.......
"settings" : {
"index" : {
"lifecycle" : {
"name" : "stephen",
"rollover_alias" : "stephen"
},
"routing" : {
"allocation" : {
"include" : {
"_tier_preference" : "data_content"
}
}
},
"number_of_shards" : "1",
"provided_name" : "",
"creation_date" : "1621703920043",
"number_of_replicas" : "1",
"uuid" : "oAMkIXECQM6qoLnnf0T1zQ",
"version" : {
"created" : "7120199"
}
}
}
```
---
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 22, 2021, 6:18pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/7 "2021-05-22T18:18:21Z")
> [@stephenb](#):
>
> ```auto
> ilm/policy/stephen
> {
> "policy": {
> "phases": {
> "hot": {
> "min_age": "0ms",
> "actions": {
> "rollover": {
> "max_size": "5gb",
> "max_age": "1d"
> }
> }
> }
> }
> }
> }
>
> ```
thanks @stephenb i will try this with filebeat configuration, can you also let me know how to write it to different indices based on source data from filebeat with different ilm rollover pattern ?
for example , i would need a filebeat output configuration like below
```
all info level messages , log goes to index called **ksenthil-info-000001**
all error level messages , log should go to **ksenthil-error-000001**
all debug level messages , log should go to **ksenthil-debug-000001**
```
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 22, 2021, 6:35pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/8 "2021-05-22T18:35:42Z")
Hmmm ... little more complex but you can figure this out.
I think Then you are going to take the ILM part out of filebeat. You will need to then setup up the boot strap index yourself (which you seem to know how to do) then use something like below.
```
setup.ilm.enabled: false
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 22, 2021, 6:49pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/9 "2021-05-22T18:49:28Z")
thanks for details @stephenb , that was a test ILM policy , will create much longer policy (56 days in production) .
i will try all these settings and let you know the outcome.
Also i have another question , like do you suggest us to use ingest pipeline or logstash pipeline ? we chose ingest pipeline as it comes along with elastic, whereas managing logstash becomes a overhead.
`Is there any important functionality today that we cant achieve with ingest pipeline , but only with logstash pipelines?`
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [May 22, 2021, 6:57pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/10 "2021-05-22T18:57:10Z")
Ok 🙂 @Kannappan_Somu That is a big question
Yes there are probably some things you can do in a logstash pipeline vs ingest pipeline. Logstash has a lot of input plugins, and I suspect there are still some filters etc.. etc but I do not know the list off hand, ingest is getting investment.
It comes down to architecture, managebility and where you want to do your processing.
I would do your research.
Good Luck!
---
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 22, 2021, 7:01pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/11 "2021-05-22T19:01:22Z")
thank you @stephenb
---
### Author:  [@Kannappan\_Somu](https://discuss.elastic.co/u/Kannappan_Somu)
#### Post date: [May 23, 2021, 2:26pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/12 "2021-05-23T14:26:15Z")
thanks @stephenb , following above configurations worked perfectly.
so filebeat with ingest pipelines to work with automated rollover , rollover settings has to be updated in filebeat output configuration.
Is there any other way to achieve the same thing by using settings in ingest pipeline , etc ?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 20, 2021, 2:26pm UTC](https://discuss.elastic.co/t/how-to-send-ingest-pipeline-output-to-a-index-alias-name/273697/13 "2021-06-20T14:26:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.