Logstash with elasticsearch input and output keep looping results forever

Elastic Stack Logstash
1

The issue is absolutely equal to Logstash with elasticsearch input and output keep looping results

I have one elastic index as input, and another as output. I've played with scroll and size options - the result is the same: docs from first index keep adding to new index non-stop.
I've started with 20 mln docs index and scroll=360m, but after ~180min index was reindexed and kept on filling with dublicated data.

I then tried with smaller indexes with 100k docs - the behaviour is the same. Should logstash even stop after the whole input index was read and reindexed, or it will run in cycle forever?

Current config is:

input {
  elasticsearch {
   hosts => ["http://111.us-east-1.elb.amazonaws.com:9200"]
   index => "first"
   size => 10000
   scroll => "360m"
  }
}

filter {
    mutate {
        convert => {
            "[query][from]" => "string"
            "[query][to]" => "string"
        }
    }
}

output {
  elasticsearch {
    hosts => ["http://111.us-east-1.elb.amazonaws.com:9200"]
    index => "second"
    sniffing => false
  }
}
1 Like
2

@OvBalaban did you manage to resolve that one somehow a year ago ? :sweat_smile:

3

@guyboertje maybe you can explain this one please ?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.