# Is it possible to let elasticsearch choose the index to ingest

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343>\
**Category:** Elasticsearch\
**Created:** [August 18, 2022, 6:23am UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343 "2022-08-18T06:23:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [August 18, 2022, 6:23am UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343/1 "2022-08-18T06:23:03Z")

</div>

Hi,  
we want to distribute / split out data to different indexes, because based on the type of the data we will have different retention times.

In my last project I used logstash to parse and enrich log events and calculated in logstash in which index the documents are stored:

```auto
output
{
  elasticsearch
  {
    ...
    index => "%{[@metadata][indexName]}"
  }
}

```

I am not familar with the ingest pipelines which are offered by elasticsearch.  
Is it possible to implement some logic like this in the ingress pipeline?

Assume logName is a field which is already present in the input documents.

```auto
if (logName in (httpd, session, xyz))
{
  store in index_a
}
else if (logName in (abc, efg)
{
  store in index b
}
else
{
  store in index c
}

```

Index a, b, c will have different ILM rules.

Is it possible to do this in elasticsearch? How?  
I'd like the centralized way to have the config in one central point. But beside my liking, what is best practice if you have multiple logstash instances?

PS: We are only using the free version. No payed subscription yet.

Thanks, Andreas

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 18, 2022, 7:52am UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343/2 "2022-08-18T07:52:30Z")

</div>

> [@asp](#):
>
> Is it possible to implement some logic like this in the ingress pipeline?

Sure is. It's done with [Pipeline processor | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/pipeline-processor.html) and having one processor per index.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [August 24, 2022, 1:38pm UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343/3 "2022-08-24T13:38:37Z")

</div>

how do I set the index / data stream to write to in a pipeline rule? Could you please provide an example?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 24, 2022, 9:20pm UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343/4 "2022-08-24T21:20:04Z")

</div>

[Ingest pipelines | Elasticsearch Guide [8.4] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html#conditionally-run-processor) is the best reference, it's similar to what you have above.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 21, 2022, 9:20pm UTC](https://discuss.elastic.co/t/is-it-possible-to-let-elasticsearch-choose-the-index-to-ingest/312343/5 "2022-09-21T21:20:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.