Why only elasticsearch container starts and kibana and filebeat containers are not getting up?

[Mahesh_Abnave]

I have an application running on six services (or containers) running on an existing docker network. I have deployed them on my local machine using docker stack deploy command. I wanted to create another set of containers of Elasticsearch, kibana and filebeat to monitor the existing docker services services. I went through some online tutorials and figured out below docker compose configuration:

docker-compose.yml

version: "3.8"

networks: 
  existing_network: 
    external: true

services:
  elasticsearch:
    image: 'docker.elastic.co/elasticsearch/elasticsearch:8.0.0'
    container_name: elasticsearch
    ports:
      - '9200:9200'
    networks:
      - existing_network

  kibana:
    image: 'docker.elastic.co/kibana/kibana:8.0.0' 
    container_name: kibana
    ports:
      - '5601:5601'
    healthcheck:
      test: ["CMD", "curl", "-f", "kibana:5601"]
      interval: 10s
      timeout: 10s
      retries: 5
    depends_on:
      - elasticsearch
    environment:
      - 'ELASTICSEARCH_HOSTS=http://elasticsearch:9200'
    networks:
      - existing_network

  filebeat:
    image: 'docker.elastic.co/beats/filebeat:8.0.0' 
    user: oem
    container_name: filebeat
    volumes:
      - /home/oem/elk/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    depends_on: 
      - elasticsearch
      - kibana
    networks:
      - existing_network

This is the filebeat configuration I tried:

filebeat.docker.yml

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'

After running

> docker stack deploy -c docker-compose.yml elk
Creating service elk_elasticsearch
Creating service elk_kibana
Creating service elk_filebeat

I tried to check kibana's logs:

> docker service logs -f elk_kibana
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | i Kibana has not been configured.
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | 
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | Go to http://0.0.0.0:5601/?code=537558 to get started.
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | 
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | 
elk_kibana.1.wvvxup2ltt9o@Mahesh-Ubuntu    | [2022-02-17T00:12:20.458+00:00][INFO ][plugins-system.preboot] Stopping all plugins.
elk_kibana.1.6i1ucgg2hzi7@Mahesh-Ubuntu    | [2022-02-17T00:13:41.748+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
elk_kibana.1.6i1ucgg2hzi7@Mahesh-Ubuntu    | [2022-02-17T00:13:41.895+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
elk_kibana.1.6i1ucgg2hzi7@Mahesh-Ubuntu    | [2022-02-17T00:13:41.951+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
elk_kibana.1.6i1ucgg2hzi7@Mahesh-Ubuntu    | [2022-02-17T00:13:41.953+00:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
elk_kibana.1.6i1ucgg2hzi7@Mahesh-Ubuntu    | [2022-02-17T00:13:41.997+00:00][INFO ][root] Holding setup until preboot stage is completed.

From the above logs, I realized that Kibana is not getting configured and is exiting. So, I was not able to open http://0.0.0.0:5601 in the browser.

I tried to check filebeat logs. But following command outputs nothing:

$docker service logs -f elk_filebeat

This no-logs behavior seem to be the bug: 1, 2 and 3.

So I tried to add the following to filebeat.docker.yml (though I didn't know if this is the correct configuration):

setup.kibana:
   host: 'kibana:5601'

But I got the same behavior: only Elasticsearch was getting up and rest were down.
Next, I tried to add following to filebeat.docker.yml:

environment:
  output.elasticsearch.hosts: '["elasticsearch:9200"]'
  setup.kibana.host: kibana:5601

Still the exact same behavior.

Q1. It seems that I am missing something basic here. How can I make all three Elasticsearch, kibana and filebeat up and running?
Q2. Also can someone point me to any tutorial(s) (possibly official ones) which discuss monitoring existing applications and infrastructure deployed on the docker containers?