File corruption occurred on recovery but checksums are ok

Hello, I'm looking for direction on this error.
Last week after cleaning and reinstanciating a cluster the error appear.
RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]

I have no clue where to begin.
The ELK stack is instanciated in a docker swarm environnement.
The IP mentionned are the overlay network from docker, if that would help.

Useful details below :

GET _cluster/allocation/explain?pretty

{
  "index" : ".ds-pfs_cachehttp-000004",
  "shard" : 1,
  "primary" : false,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "ALLOCATION_FAILED",
    "at" : "2021-09-06T11:30:50.408Z",
    "failed_allocation_attempts" : 5,
    "details" : "failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ",
    "last_allocation_status" : "no_attempt"
  },
  "can_allocate" : "no",
  "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
  "node_allocation_decisions" : [
    {
      "node_id" : "55yKGeAvTQOeAeXkwKsrrw",
      "node_name" : "es-data-hot-z1-1-p9cf9jinorunhzipsu0jp4hev",
      "transport_address" : "10.0.34.34:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179869184",
        "rack" : "zone1",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-06T11:30:50.408Z], failed_attempts[5], failed_nodes[[6cFblf1ISzC7Bc46G9V-0w, zG1R0rbIQl6X8Bt3A9tczw, 55yKGeAvTQOeAeXkwKsrrw]], delayed=false, details[failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], allocation_status[no_attempt]]]"
        }
      ]
    },
    {
      "node_id" : "6cFblf1ISzC7Bc46G9V-0w",
      "node_name" : "es-data-hot-z1-2-p9cf9jinorunhzipsu0jp4hev",
      "transport_address" : "10.0.34.47:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179869184",
        "rack" : "zone1",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-06T11:30:50.408Z], failed_attempts[5], failed_nodes[[6cFblf1ISzC7Bc46G9V-0w, zG1R0rbIQl6X8Bt3A9tczw, 55yKGeAvTQOeAeXkwKsrrw]], delayed=false, details[failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], allocation_status[no_attempt]]]"
        }
      ]
    },
    {
      "node_id" : "8CCfzfIlQfmGWK7VdYpWGw",
      "node_name" : "es-data-hot-z2-1-8c8ds36js47jcu6a9p5juz90a",
      "transport_address" : "10.0.34.38:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179869184",
        "rack" : "zone2",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-06T11:30:50.408Z], failed_attempts[5], failed_nodes[[6cFblf1ISzC7Bc46G9V-0w, zG1R0rbIQl6X8Bt3A9tczw, 55yKGeAvTQOeAeXkwKsrrw]], delayed=false, details[failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], allocation_status[no_attempt]]]"
        },
        {
          "decider" : "awareness",
          "decision" : "NO",
          "explanation" : "there are too many copies of the shard allocated to nodes with attribute [rack], there are [2] total configured shard copies for this shard id and [3] total attribute values, expected the allocated shard count per attribute [2] to be less than or equal to the upper bound of the required number of shards per attribute [1]"
        }
      ]
    },
    {
      "node_id" : "IhnjpdG-Rbq_n0caOqOlCQ",
      "node_name" : "es-data-hot-z1-1-2e7hi8dql06atwgh5pu05bqmp",
      "transport_address" : "10.0.34.32:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179869184",
        "rack" : "zone1",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-06T11:30:50.408Z], failed_attempts[5], failed_nodes[[6cFblf1ISzC7Bc46G9V-0w, zG1R0rbIQl6X8Bt3A9tczw, 55yKGeAvTQOeAeXkwKsrrw]], delayed=false, details[failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], allocation_status[no_attempt]]]"
        }
      ]
    },
    {
      "node_id" : "M_HG7azrRASwJo5xVRk-gw",
      "node_name" : "es-data-hot-z1-1-qz8oolh9idypl5vod0raah86k",
      "transport_address" : "10.0.34.33:9300",
      "node_attributes" : {
        "ml.machine_memory" : "17179869184",
        "rack" : "zone1",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true",
        "transform.node" : "true"
      },
      "node_decision" : "no",
      "deciders" : [
        {
          "decider" : "max_retry",
          "decision" : "NO",
          "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2021-09-06T11:30:50.408Z], failed_attempts[5], failed_nodes[[6cFblf1ISzC7Bc46G9V-0w, zG1R0rbIQl6X8Bt3A9tczw, 55yKGeAvTQOeAeXkwKsrrw]], delayed=false, details[failed shard on node [zG1R0rbIQl6X8Bt3A9tczw]: failed recovery, failure RecoveryFailedException[[.ds-pfs_cachehttp-000004][1]: Recovery failed from {es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8}{OmvEo07cTTegiaRlL36KVg}{p6ODtzb5QP-xQu9vR5FWYA}{10.0.34.36}{10.0.34.36:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone2, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {es-data-hot-z1-2-2e7hi8dql06atwgh5pu05bqmp}{zG1R0rbIQl6X8Bt3A9tczw}{X4nnGcNsSOqw2tmJAS3Qvw}{10.0.34.45}{10.0.34.45:9300}{cdhilrstw}{ml.machine_memory=17179869184, rack=zone1, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; nested: RemoteTransportException[[es-data-hot-z2-1-x0hxwanma57yajfoe61amoek8][10.0.34.36:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[File corruption occurred on recovery but checksums are ok]]; ], allocation_status[no_attempt]]]"
        }
      ]
    },
...

GET _cluster/stats?human&pretty

{
  "_nodes" : {
    "total" : 20,
    "successful" : 20,
    "failed" : 0
  },
  "cluster_name" : "cachehttp-prod",
  "cluster_uuid" : "vgAHVoTTQoCieDO_atFr6A",
  "timestamp" : 1631002552613,
  "status" : "red",
  "indices" : {
    "count" : 25,
    "shards" : {
      "total" : 97,
      "primaries" : 44,
      "replication" : 1.2045454545454546,
      "index" : {
        "shards" : {
          "min" : 2,
          "max" : 12,
          "avg" : 3.88
        },
        "primaries" : {
          "min" : 1,
          "max" : 6,
          "avg" : 1.76
        },
        "replication" : {
          "min" : 0.8,
          "max" : 11.0,
          "avg" : 1.392
        }
      }
    },
    "docs" : {
      "count" : 1522718145,
      "deleted" : 140408
    },
    "store" : {
      "size" : "1tb",
      "size_in_bytes" : 1150170071017,
      "reserved" : "0b",
      "reserved_in_bytes" : 0
    },
    "fielddata" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "evictions" : 0
    },
    "query_cache" : {
      "memory_size" : "0b",
      "memory_size_in_bytes" : 0,
      "total_count" : 0,
      "hit_count" : 0,
      "miss_count" : 0,
      "cache_size" : 0,
      "cache_count" : 0,
      "evictions" : 0
    },
    "completion" : {
      "size" : "0b",
      "size_in_bytes" : 0
    },
    "segments" : {
      "count" : 1922,
      "memory" : "19mb",
      "memory_in_bytes" : 20017280,
      "terms_memory" : "14.1mb",
      "terms_memory_in_bytes" : 14837568,
      "stored_fields_memory" : "1.3mb",
      "stored_fields_memory_in_bytes" : 1438768,
      "term_vectors_memory" : "0b",
      "term_vectors_memory_in_bytes" : 0,
      "norms_memory" : "15kb",
      "norms_memory_in_bytes" : 15360,
      "points_memory" : "0b",
      "points_memory_in_bytes" : 0,
      "doc_values_memory" : "3.5mb",
      "doc_values_memory_in_bytes" : 3725584,
      "index_writer_memory" : "659.5mb",
      "index_writer_memory_in_bytes" : 691566580,
      "version_map_memory" : "12.1mb",
      "version_map_memory_in_bytes" : 12740868,
      "fixed_bit_set" : "280.1kb",
      "fixed_bit_set_memory_in_bytes" : 286904,
      "max_unsafe_auto_id_timestamp" : 1630972829582,
      "file_sizes" : { }
    },
    "mappings" : {
      "field_types" : [
        {
          "name" : "binary",
          "count" : 3,
          "index_count" : 1
        },
        {
          "name" : "boolean",
          "count" : 50,
          "index_count" : 17
        },
        {
          "name" : "date",
          "count" : 91,
          "index_count" : 22
        },
        {
          "name" : "double",
          "count" : 4,
          "index_count" : 4
        },
        {
          "name" : "flattened",
          "count" : 1,
          "index_count" : 1
        },
        {
          "name" : "float",
          "count" : 54,
          "index_count" : 9
        },
        {
          "name" : "half_float",
          "count" : 44,
          "index_count" : 12
        },
        {
          "name" : "integer",
          "count" : 114,
          "index_count" : 10
        },
        {
          "name" : "keyword",
          "count" : 806,
          "index_count" : 24
        },
        {
          "name" : "long",
          "count" : 918,
          "index_count" : 23
        },
        {
          "name" : "nested",
          "count" : 31,
          "index_count" : 10
        },
        {
          "name" : "object",
          "count" : 698,
          "index_count" : 24
        },
        {
          "name" : "short",
          "count" : 12,
          "index_count" : 4
        },
        {
          "name" : "text",
          "count" : 118,
          "index_count" : 11
        }
      ]
    },
    "analysis" : {
      "char_filter_types" : [ ],
      "tokenizer_types" : [ ],
      "filter_types" : [ ],
      "analyzer_types" : [ ],
      "built_in_char_filters" : [ ],
      "built_in_tokenizers" : [ ],
      "built_in_filters" : [ ],
      "built_in_analyzers" : [ ]
    }
  },
  "nodes" : {
    "count" : {
      "total" : 20,
      "coordinating_only" : 0,
      "data" : 12,
      "data_cold" : 12,
      "data_content" : 12,
      "data_hot" : 12,
      "data_warm" : 12,
      "ingest" : 17,
      "master" : 3,
      "ml" : 20,
      "remote_cluster_client" : 20,
      "transform" : 12,
      "voting_only" : 0
    },
    "versions" : [
      "7.10.2"
    ],
    "os" : {
      "available_processors" : 1920,
      "allocated_processors" : 1920,
      "names" : [
        {
          "name" : "Linux",
          "count" : 20
        }
      ],
      "pretty_names" : [
        {
          "pretty_name" : "Ubuntu 20.04.2 LTS",
          "count" : 20
        }
      ],
      "mem" : {
        "total" : "234.5gb",
        "total_in_bytes" : 251792457728,
        "free" : "20gb",
        "free_in_bytes" : 21482180608,
        "used" : "214.4gb",
        "used_in_bytes" : 230310277120,
        "free_percent" : 9,
        "used_percent" : 91
      }
    },
    "process" : {
      "cpu" : {
        "percent" : 21
      },
      "open_file_descriptors" : {
        "min" : 1110,
        "max" : 1441,
        "avg" : 1252
      }
    },
    "jvm" : {
      "max_uptime" : "5d",
      "max_uptime_in_millis" : 433343179,
      "versions" : [
        {
          "version" : "11.0.10",
          "vm_name" : "OpenJDK 64-Bit Server VM",
          "vm_version" : "11.0.10+9",
          "vm_vendor" : "AdoptOpenJDK",
          "bundled_jdk" : true,
          "using_bundled_jdk" : false,
          "count" : 20
        }
      ],
      "mem" : {
        "heap_used" : "52.8gb",
        "heap_used_in_bytes" : 56717845776,
        "heap_max" : "114gb",
        "heap_max_in_bytes" : 122478264320
      },
      "threads" : 6399
    },
    "fs" : {
      "total" : "99.1tb",
      "total_in_bytes" : 109062744473600,
      "free" : "95.4tb",
      "free_in_bytes" : 104954523590656,
      "available" : "90.4tb",
      "available_in_bytes" : 99456634167296
    },
    "plugins" : [
      {
        "name" : "opendistro_security",
        "version" : "1.13.1.0",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "Provide access control related features for Elasticsearch 7",
        "classname" : "com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin",
        "extended_plugins" : [ ],
        "has_native_controller" : false
      },
      {
        "name" : "opendistro-job-scheduler",
        "version" : "1.13.0.0",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "Open Distro for Elasticsearch job schduler plugin",
        "classname" : "com.amazon.opendistroforelasticsearch.jobscheduler.JobSchedulerPlugin",
        "extended_plugins" : [ ],
        "has_native_controller" : false
      },
      {
        "name" : "opendistro-anomaly-detection",
        "version" : "1.13.0.0",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "Amazon opendistro elasticsearch anomaly detector plugin",
        "classname" : "com.amazon.opendistroforelasticsearch.ad.AnomalyDetectorPlugin",
        "extended_plugins" : [
          "lang-painless",
          "opendistro-job-scheduler"
        ],
        "has_native_controller" : false
      },
      {
        "name" : "opendistro-reports-scheduler",
        "version" : "1.13.0.0",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "Scheduler for ODFE Kibana Reports Plugin",
        "classname" : "com.amazon.opendistroforelasticsearch.reportsscheduler.ReportsSchedulerPlugin",
        "extended_plugins" : [
          "opendistro-job-scheduler"
        ],
        "has_native_controller" : false
      },
      {
        "name" : "repository-s3",
        "version" : "7.10.2",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "The S3 repository plugin adds S3 repositories",
        "classname" : "org.elasticsearch.repositories.s3.S3RepositoryPlugin",
        "extended_plugins" : [ ],
        "has_native_controller" : false
      },
      {
        "name" : "opendistro-alerting",
        "version" : "1.13.1.0",
        "elasticsearch_version" : "7.10.2",
        "java_version" : "1.8",
        "description" : "Amazon OpenDistro alerting plugin",
        "classname" : "com.amazon.opendistroforelasticsearch.alerting.AlertingPlugin",
        "extended_plugins" : [
          "lang-painless"
        ],
        "has_native_controller" : false
      }
    ],
    "network_types" : {
      "transport_types" : {
        "com.amazon.opendistroforelasticsearch.security.ssl.http.netty.OpenDistroSecuritySSLNettyTransport" : 20
      },
      "http_types" : {
        "com.amazon.opendistroforelasticsearch.security.http.OpenDistroSecurityHttpServerTransport" : 20
      }
    },
    "discovery_types" : {
      "zen" : 20
    },
    "packaging_types" : [
      {
        "flavor" : "default",
        "type" : "deb",
        "count" : 20
      }
    ],
    "ingest" : {
      "number_of_pipelines" : 2,
      "processor_stats" : {
        "gsub" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        },
        "script" : {
          "count" : 0,
          "failed" : 0,
          "current" : 0,
          "time" : "0s",
          "time_in_millis" : 0
        }
      }
    }
  }
}

Welcome to our community! :smiley:

Have you tried a plain reroute request to get it to try again? Cluster reroute API | Elasticsearch Guide [7.14] | Elastic

It looks like you're using the OpenDistro fork? Best to ask the OpenDistro folks for help instead, or else demonstrate that the Elasticsearch fork has the same problem. The two versions are different in ways that might explain this exception I think, and the OpenDistro folks don't really answer questions on these forums.

Thank you, we will investigate on this side. It seems to be a transport issue.