Script template to manipulate records using parameters

Elastic Stack Elasticsearch
1

Hi,
actually, into different indexes, I have duplicated records which have the same field's values but, obviously, different _id value.
so, trying to remove these duplicated records this is what I would like to do:

  1. query a specific index
  2. compare each found record with the others located inside index above
  3. if myfield record's value1 is equal to myfield record's value2 drop record1

essentially, writing this procedure in a generic scripting language it will looks like:

foreach element into index.2017.06.6 {
     do {
        if element.myvalue equal element[+1].myvalue{
           delete element    
         }
      }
}

can someone help me? I don't want the code to do that, I would like to have some guidelines to understand the mechanism behind the execution of scripts and how can I pass parameters inside my search.

thanks

2

You won't be able to do this within a search script, since the script can only decide if a document matches the search, not try to remove a document at the same time. I think you will need to do this from your client. I don't think you will need an elasticsearch script though. It should just be a normal query, look at the results with the pseudocode you already have above, and if you find the condition you are looking for, issue a delete request for that document.

3

thank you for the answer. one doubt? what do you mean with "...you will need to do this from your client" ?
which client?

4

I mean from your application, which is using some client (whether eg an elasticsearch client or using http directly).

5

I feel myself really stupid.
actually this is my structure

  1. one script is extracting data to fill a file called MyFile
  2. filebeat reads MyFile
  3. filebeat sends read records to Logstash
  4. Logstash normalize them and sends them to Elastic
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.