Metricbeat Kibana Dashboard Request Time-out when there is a Virtual IP in the VM being monitored

[thesn]

hi,

I have ES 7.11.1 and Kibana 7.11.1 setup in docker container (10.6.226.80) using docker-compose in a Linux VM.
Also, I have Metricbeat 7.11.1 setup in docker container (10.6.226.104) using docker-compose in a Linux VM.

I set the setup.dashboards.enable: true
When I start the Metricbeat (docker-compose up -d metricbeat), I saw error in the logs:

    2021-03-01T09:39:31.328Z	ERROR	instance/beat.go:971	Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 408 to import file: <nil>. Response: {"statusCode":408,"error":"Request Time-out","message":"Request Time-out"}
Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: failed to import Kibana index pattern: 1 error: error loading index pattern: returned 408 to import file: <nil>. Response: {"statusCode":408,"error":"Request Time-out","message":"Request Time-out"}

After comparing with other VMs (which the Metricbeat is running with no issues), I found out that the VM (10.6.226.104) has a Virtual IP (10.6.226.250) up in the same network interface.
This Virtual IP is used for database fail-over. Thus, it has to stay there.
This Virtual IP is my main suspect of what causing the Request Time-out error.

Any suggestion on how to still be able to monitor this VM using Metricbeat?

Thank you.

my docker-compose:

    version: "2.2"
services:
  metricbeat:
    image: registry.bankbjb.co.id/metricbeat:7.11.1
    container_name: metricbeat
    hostname: bjbmdwdbdev
    user: root #to read the docker socket
    volumes:
      - "./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro"
      - "./metricbeat/fields.yml:/usr/share/metricbeat/fields.yml:ro"
      - "/proc:/hostfs/proc:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/:/hostfs:ro"
    command: metricbeat -e -system.hostfs=/hostfs -strict.perms=false

my metricbeat.yaml:

    # =========================== Modules configuration ============================
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true

metricbeat.modules:
- module: system
  metricsets: ["cpu","memory","network","filesystem"]
  enabled: true
  period: 60s
  processes: ['.*']
  cpu.metrics:  ["percentages","normalized_percentages"]
  core.metrics: ["percentages"]

  process.include_top_n:
    enabled: true
    by_cpu: 10
    by_memory: 10

# ================================= Dashboards =================================
setup.dashboards.enabled: true

# ================================== Logging ===================================
logging.metrics.period: 60s

# ================================== Template ==================================
setup.template.fields: "fields.yml"

# Elasticsearch template settings
setup.template.settings:
  index:
    number_of_shards: 1
    codec: best_compression

# =================================== Kibana ===================================
setup.kibana:
  host: "10.6.226.80:5601"
  space.id: "system-monitor"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  enabled: true
  hosts: ["10.6.226.80:9200"]
  protocol: "http"