# Elastic, spark and ILM

**URL:** <https://discuss.elastic.co/t/elastic-spark-and-ilm/228750>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management\
**Created:** [April 19, 2020, 5:57pm UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750 "2020-04-19T17:57:44Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Martin\_Farmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_farmer/32/64456_2.png) [@Martin\_Farmer](https://discuss.elastic.co/u/Martin_Farmer)\
**Post date:** [April 19, 2020, 5:57pm UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750/1 "2020-04-19T17:57:44Z")

</div>

So - I have filebeat working great elastic and ILM, and separately I have my scala/spark code feeding elastic with custom documents - and that works great. What I can't figure out or find documentation on, is how to integrate the spark documents with the ILM - Any pointers or help would be greatly appreciated!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 19, 2020, 11:15pm UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750/2 "2020-04-19T23:15:03Z")

</div>

The only integration you should need to do there is send the documents to the alias. This would be once you've setup the ILM policy, which might be what you're looking to do in spark?

---

<div class="post-metadata">

**Author:** ![Martin\_Farmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_farmer/32/64456_2.png) [@Martin\_Farmer](https://discuss.elastic.co/u/Martin_Farmer)\
**Post date:** [April 19, 2020, 11:16pm UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750/3 "2020-04-19T23:16:55Z")

</div>

to clarify - I write to the index named by the alias ? I thought the ILM picked up the template from the matching index name ?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 20, 2020, 12:13am UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750/4 "2020-04-20T00:13:20Z")

</div>

You write to the alias itself, not the underlying index, as that will change when ILM rotates it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2020, 12:13am UTC](https://discuss.elastic.co/t/elastic-spark-and-ilm/228750/5 "2020-05-18T00:13:24Z")

</div>

Closed on May 18, 2020, 12:13am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.