Move documents from one index to another

Elastic Stack Elasticsearch
1

Hi guys,

I have the index where I store data. My question is whether I can restore the data that meet some conditions to a new index by using script reindexing so I can visualize the data properly.

Basically I'm thinking about something like that:

"script": { 
    "source":  "if id has suffix = "ap" && suffix = "bn" within 60 mins move data from index1 to index2"
}

If I cannot do it with reindexing what can I use instead?

Thank you for any advice :slight_smile:

2

I'm sorry, I'm not sure I understand the requirement fully. You can use scripts, for suffix/prefix checks on the id, but that 60 minute check I am not sure about. There is no notion of a cronjob for this, but you can trigger this manually every 60 minutes, and filter documents based on a timestamp?

3

Hey @spinscale, thank you for your reply! I'm struggling with this a while :sweat_smile:

In particular, we have a tons of log messages constantly coming in the following format:

Jul 23 09:24:16 mmr mmr-core[5147]:  Aweg3AOMTs_1563866656876839.mt
Jul 23 09:24:18 mmr mmr-core[5210]:  Aweg3AOMTs_1563866656876839.0.dn

There are different id numbers (1563866656876839) and two possible suffixes (mt/dn).

We parse it with logstash and store these messages in one index.

When the id number with mt suffix gots dn suffix within 1 hour it means GOOD and it should get a new field status with approved value in it. If not the field value should be disapproved.

So in the end a new index isn't needed :smiley: But I'm still curious how to achieve that and if it is even possible to create and fill the new field in document based on a time condition or how to say...

4

Hey,

I see. So what I think you want is to create a connection between two log lines, like capturing start and end.

If you can parse the logline so that Aweg3AOMTs_1563866656876839 becomes it's own field, you could maybe take a look at data frames to create a summary index, that contains information if a document has a start and end time and thus is in a good state.

See https://www.elastic.co/guide/en/elasticsearch/reference/7.3/transforms.html

--Alex

5

Hey, thanks for help!

I see. So what I think you want is to create a connection between two log lines, like capturing start and end.

I think that is what we need. We have loglines parsed already like that:

messageId: Aweg3AOMTs_1563866656871111.0.dn
text: Aweg3AOMTs
num1: 1563866656871111
num2: 0
suffix: mt/dn

And I have the logs in dataframes grouped by num1. Is the feature you are talking about free or part of paid X-Pack?

I'm not sure if I understand correctly the part about start and end time of a document and how is possible to give a document a state?

Thank you very much for any further clarification!

6

that is part of basic and thus free to use.