Getting system metrics from metricbeat in docker container

Elastic Stack Beats
,
1

I'm trying to implement a metricbeat service running from a container in order to ingest host metrics into my elasticsearch cluster.

On each server, I have a metricbeat running as a service directly on the host and the idea is to migrate it to a containerised version. My idea is to make a migration without much data loss, so I am not interested in shutting down these services on the host until the container version is working properly.

So far, I have only tested with system metrics, for the ones that are defined with period (1m and 15m) everything works correctly. When i want to ingest metrics from the metricset process, I find that only data from the container processes are being saved and not from the host.

My problem is the following, according to the following post, it is necessary to assign the host network to the container, but only for the network metricset to work properly that is not my case.

I understand that when i want to collect network metrics i will need to do that configuration. But for the moment i don't see it necessary. Now, in case it is necessary to assign the host network, what problems could i have in networking issues due to how my elasticsearch cluster is configured? The idea is not to suffer any outage.

Just in case it is useful, I have an elasticsearch cluster in production deployed on two different servers, all services are dockerised. On server 1, there are two nodes (node1 and node2). On server 2, there is only one node (node3). Each server has 2 network interfaces, so, in order to establish communication between the 3 nodes, I configured the networking options of the nodes as follows:

All the services (elasticsearch, kibana, metricbeat) runs in 7.17.9 version.

server 1 - docker-compose.yml
version: '3.7'

services:
  node01:
    ports:
      - "ip-addr-1:9200:9200"
      - "ip-addr-1:9300:9300"

  node02:
    ports:
      - "ip-addr-1:9200:9200"
      - "ip-addr-1:9300:9300"
server 2 - docker-compose.yml
version: '3.7'

services:
  node01:
    ports:
      - "ip-addr-1:9200:9200"
      - "ip-addr-1:9300:9300"

Here are metricbeat configurations used.

docker-compose.yml
version: '3.7'

services:
  
  metricbeat01:
    image: docker.elastic.co/beats/metricbeat:7.17.9
    container_name: metricbeat01
    restart: on-failure
    user: root  
    volumes: 
      - ./config/enable-modules.sh:/usr/share/metricbeat/enable-modules.sh:ro      
      - ./config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro           
      - ./config/modules.d/:/usr/share/metricbeat/modules.d/:ro                   
      - /var/run/docker.sock:/var/run/docker.sock:ro                            
      - /proc:/hostfs/proc:ro                                              
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
    command: ["/bin/bash", "-c", "/usr/share/metricbeat/enable-modules.sh && metricbeat -e -d \"*\""]
    networks:
      - docker_default

networks:
  docker_default:
    name: docker_default
    external: true
metricbeat.yml
metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.period: 60s
  reload.enabled: true

setup.template.enabled: false
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

output.elasticsearch:
  hosts: ["ip-addr-node2:9200"]
  index: "%metricbeat-%testing-%{[agent.version]}-%{+yyyy.MM.dd}" 
  username: "elastic"
  password: "secret"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
modules.d/system.yml
- module: system
  period: 10s
  metricsets:
    #- cpu
    #- load
    #- memory
    #- network
    - process
    #- process_summary
    #- socket_summary
    #- entropy 
    #- core    
    #- diskio  
    #- socket  
    #- service 
    #- users   
  process.include_top_n:
    by_cpu: 5
    by_memory: 5
  system.hostfs: "/hostfs"

# - module: system
#   period: 1m
#   metricsets:
#     - filesystem 
#     - fsstat

# - module: system
#   period: 15m
#   metricsets:
#     - uptime