So - I have filebeat working great elastic and ILM, and separately I have my scala/spark code feeding elastic with custom documents - and that works great. What I can't figure out or find documentation on, is how to integrate the spark documents with the ILM - Any pointers or help would be greatly appreciated!
The only integration you should need to do there is send the documents to the alias. This would be once you've setup the ILM policy, which might be what you're looking to do in spark?
to clarify - I write to the index named by the alias ? I thought the ILM picked up the template from the matching index name ?
You write to the alias itself, not the underlying index, as that will change when ILM rotates it.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.