Schedule a Job in Elasticsearch

Elastic Stack Elasticsearch
1

Hi,
is it possible to schedule a job in elasticsearch and save it?

My requirements are that the team only wants to keep certain messages after xy days to reduce the storage capacity.

Can i realize it with rollover, ilm? I only thought about reindex and then delete the old index. But I didnt found a way to schedule jobs/tasks or store it somewhere. And we dont want to create another container which only runs this one job via the api.

2

I believe that ILM is designed for that. (Index Lifecycle Management)

3

I use ILM but i didnt find a way to give ILM more instructions to only delete xy messages and keep others.

4

If you are using time based indices, which is the recommendation for this kind of use case, then it should be easy to configure ILM to move old indices to cold or delete phase...

5

Yes this is pretty easy, but i want to delete only specific messages from indices not the complete index.

6

Deleting documents from indices is very expensive and inefficient compared to deleting complete indices so is not recommended nor supported by ILM. You will therefore need to set up a cron job or similar to periodically run a delete by query.

7

Yeah i know this too, this is why I thought if i can maybe reindex only the wanted messages into a new index and delete the old one with some form of job in elasticsearch.

It would be a reindex and one delete job. or only the reindex and delete via ILM. Is this possible to store some job somewhere like GitHub - opendistro-for-elasticsearch/job-scheduler: 🕓 Open Distro for Elasticsearch Job Scheduler ?

8

For that you probably need to ask in the OpenDistro forum.

9

@Christian_Dahlqvist I know i dont get infos about it here, but i used it as a comparison.... because i didnt found anything like it. And I am not interested in the opendistro elasticsearch fight...

Is there a function in elasticsearch to store tasks which are executed in a periodic way?

10

Not that I know of.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.