Query field value to reindex existing indexes

Hello,

I'm trying to achieve a complex action where we currently have a common index filebeat-%{beat version}-%{YY-mm-dd} type of index with multiple log.file.path different. The idea is to have them in other index depending on this log.file.path.

Ex:
From:

filebeat-7.17.1-2022.01
      log.file.path: /var/log/log1
      log.file.path: /var/log/log2
      log.file.path: /var/log/log3

To:

filebeat-log1-2022.01
      log.file.path: /var/log/log1
filebeat-log2-2022.01
      log.file.path: /var/log/log2
filebeat-log3-2022.01
      log.file.path: /var/log/log3

Here I tried with the following setup in logstash to try

input {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "filebeat-7*"
    query => '{
      "query":{
          "match":{
              "log.file.path" : "/var/log/log1"
          }
        }
    }'
    size => 10000
    scroll => "20s"
  }
}

output {
  elasticsearch {
      hosts => "http://localhost:9200"
      index => "filebeat-log1-%{+YYYY.MM}"
    }
}
input{
  elasticsearch {
    hosts => ["http://localhost:9200"]
  }
}

filter {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => ["filebeat-7*"]
    query => "log.file.path:/var/log/log1"
  }
}

output {
  elasticsearch {
      hosts => "http://localhost:9200"
      index => "filebeat-log1-%{+YYYY.MM}"
    }
}

In my first setup, index are created correctly for other log comes into picture and in the second case, nothing is done.

Do you have any suggestion on this topic ?

Thank you in advance for your help.

Best regards,
Benjamin

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.