Continuously copy specific documents to another index?

Elastic Stack Elasticsearch
1

Hi All,

we're using Elasticsearch for storing our application logs using daily indexes, which we delete after two weeks. Now we would like to keep all logs with the loglevel "error" for another 2 months.

We thought about using a separate index for all error messages and apply the different deletion rules there.

However we don't want to add the logic regarding in which index to write ("normal" or the ones for the errors) in each of our app servers logstash configs.

Is there a way to stream / continuously copy specific documents (the ones with level "error") from one index to another?

It seems like the ReIndex API could be used for that.

  • Is the right way to schedule a job which runs every 5 minutes or so with the ReIndex command?
  • Does ReIndex check which documents already exist and skips those?
  • If we run the script every 5 mins, we probably can also add a "query" field to limit the documents compared to the last 5 mins.

Or is there a better way? :slight_smile:

Thanks!

Thomas

2

You should be able to do this with Alerting, using a search input and an index action.
That way you can schedule it to run once a day, for eg.

3

Thanks for the reply! Unfortunately we only have the open source version for various reasons, is there any way to do it without the X-Pack features?

4

That'd have to be a reindex with an external trigger.

5

Ok, so we would setup a script to run every few mins, call ReIndex with a query to only match the errors.

Two last questions:

  • Will ReIndex make sure matching documents are not copied twice? Thats what I understand from the docs.
  • We should probably also include the timestamp in the query to limit the results to the last X minutes (basically the script interval + a few mins).

Thanks a lot for your help!

6

Can you link to that?

Yes, ideally.

7

Setting version_type to external causes Elasticsearch to preserve the version from the source, create any documents that are missing, and update any documents that have an older version in the destination than they do in the source.

I understand that this means, that documents which already exist in the target index will be updated if there is a newer version available, but not inserted twice.

8

Yep!

9

I'll try it out, thanks for your help! :+1:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.