Move documents from one index to another

Hi guys,

I have the index where I store data. My question is whether I can restore the data that meet some conditions to a new index by using script reindexing so I can visualize the data properly.

Basically I'm thinking about something like that:

"script": { 
    "source":  "if id has suffix = "ap" && suffix = "bn" within 60 mins move data from index1 to index2"

If I cannot do it with reindexing what can I use instead?

Thank you for any advice :slight_smile:

I'm sorry, I'm not sure I understand the requirement fully. You can use scripts, for suffix/prefix checks on the id, but that 60 minute check I am not sure about. There is no notion of a cronjob for this, but you can trigger this manually every 60 minutes, and filter documents based on a timestamp?

Hey @spinscale, thank you for your reply! I'm struggling with this a while :sweat_smile:

In particular, we have a tons of log messages constantly coming in the following format:

Jul 23 09:24:16 mmr mmr-core[5147]:
Jul 23 09:24:18 mmr mmr-core[5210]:  Aweg3AOMTs_1563866656876839.0.dn

There are different id numbers (1563866656876839) and two possible suffixes (mt/dn).

We parse it with logstash and store these messages in one index.

When the id number with mt suffix gots dn suffix within 1 hour it means GOOD and it should get a new field status with approved value in it. If not the field value should be disapproved.

So in the end a new index isn't needed :smiley: But I'm still curious how to achieve that and if it is even possible to create and fill the new field in document based on a time condition or how to say...


I see. So what I think you want is to create a connection between two log lines, like capturing start and end.

If you can parse the logline so that Aweg3AOMTs_1563866656876839 becomes it's own field, you could maybe take a look at data frames to create a summary index, that contains information if a document has a start and end time and thus is in a good state.



Hey, thanks for help!

I see. So what I think you want is to create a connection between two log lines, like capturing start and end.

I think that is what we need. We have loglines parsed already like that:

messageId: Aweg3AOMTs_1563866656871111.0.dn
text: Aweg3AOMTs
num1: 1563866656871111
num2: 0
suffix: mt/dn

And I have the logs in dataframes grouped by num1. Is the feature you are talking about free or part of paid X-Pack?

I'm not sure if I understand correctly the part about start and end time of a document and how is possible to give a document a state?

Thank you very much for any further clarification!

that is part of basic and thus free to use.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.