Data missing with logstash elasticsearch-input-plugin

[higee]

Hi,

I was using Logstash (7.4.2) to transfer documents between Elasticsearch indices (7.4.2). First have a look at the Logstash.conf

input {
  elasticsearch {
    hosts => ""
    index => "haproxy-*"
    scroll => "10m"
    size => 10000
    query => '{
      "_source" : [
        "@timestamp", "http_status_code", 
       "request_header_forwarded_for", "http_verb",
      "request_header_user_agent"
      ],
      "query" : {
        "bool" : {
          "must": [
            {"exists": {"field": "request_header_user_agent"}},
            {"exists": {"field": "r_time"}},
          ],
          "must_not" : [
            {"term": {"http_status_code": "417"}}
          ]
        }
      }
    }'
  }
}

filter {
  mutate {
    add_field => { "target_index" => "%{[@metadata][_index]}" }
  }
  mutate {
    gsub => [
      "target_index", "service_a_haproxy_", "service_a_copy_haproxy_"
    ]
    remove_field => ["@version"]
  }
}

output {
  elasticsearch {
    hosts => ""
    index => "%{target_index}"
    document_id => "%{[@metadata][_id]}"
  }
}

As you can see I'm inheriting index and document ID of the source indices. For instance,
the service_a_haproxy-2020.02.01 index becomes service_a_copy_haproxy_2020.02.01 and document ID stays the same to be consistent between indices.

However, when I checked the result, there was a gap in the number of documents between source and target indices.

• Source
  

• Target
  

Anyone has an idea what went wrong?