Metricbeat gives me certificate issue

[adityak248 (Karthik)]

Hello Team,

I have been trying to configure the metricbeat over SSL and i keep getting the following error

metricbeat_1  | Exiting: error initializing publisher: 2 errors: open $CERTS_DIR/elk/elk.crt: no such file or directory $CERTS_DIR/elk/elk.crt; open $CERTS_DIR/ca/ca.crt: no such file or directory reading $CERTS_DIR/ca/ca.crt

I have added the cert volume to the metricbeat container as well. Please let me know if there is anything that should be done more.

Here are my configs
metricbeat.yml

#-------------------------------- Autodiscovery -------------------------------
# Autodiscover allows you to detect changes in the system and spawn new modules as they happen.
metricbeat.autodiscover:
  providers:
    - type: docker
      # https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html
      hints.enabled: true

metricbeat.modules:
#------------------------------- System Module -------------------------------
- module: system
  metricsets: ["cpu", "load", "memory", "network", "process", "process_summary", "core", "diskio", "socket"]
  processes: ['.*']
  process.include_top_n:
    by_cpu: 5
    by_memory: 5
  period: 10s
  cpu.metrics:  ["percentages"]
  core.metrics: ["percentages"]

- module: system
  period: 1m
  metricsets:
    - filesystem
    - fsstat
  processors:
  - drop_event.when.regexp:
      system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'

- module: system
  period: 15m
  metricsets:
    - uptime

#------------------------------- Docker Module -------------------------------
- module: docker
  metricsets: ["container", "cpu", "diskio", "healthcheck", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s

#================================ Processors ===================================
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_locale:
    format: offset
- add_host_metadata:
    netinfo.enabled: true

#========================== Elasticsearch output ===============================
output.elasticsearch:
  hosts: "elk:9200"
  protocol: "https"
  username: $ELASTICSEARCH_USERNAME
  password: $ELASTICSEARCH_PASSWORD
  ssl.certificate_authorities: ["$CERTS_DIR/ca/ca.crt"]
  ssl.certificate: "$CERTS_DIR/elk/elk.crt"
  ssl.key: "$CERTS_DIR/elk/elk.key"

#============================== Dashboards =====================================
setup.dashboards:
  enabled: true

#============================== Kibana =========================================
setup.kibana:
  host: "kibana:5601"
  protocol: "https"
  username: $ELASTICSEARCH_USERNAME
  password: $ELASTICSEARCH_PASSWORD
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["$CERTS_DIR/ca/ca.crt"]
  ssl.certificate: "$CERTS_DIR/elk/elk.crt"
  ssl.key: "$CERTS_DIR/elk/elk.key"

#============================== Logging =========================================

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/metricbeat
  name: metricbeat
  rotateeverybytes: 104857600 # = 100MB
  keepfiles: 3
  permissions: 0600
#============================== Xpack Monitoring ===============================
xpack.monitoring:
  enabled: true
  elasticsearch:

docker-compose.yaml

metricbeat:
    hostname: metricbeat
    user: root #To read the docker socket
    image: docker.elastic.co/beats/metricbeat:$VERSION
    logging:
      options:
        max-file: "3"
        max-size: "50m"
  # cap_add:
  #    - AUDIT_CONTROL
  #    - AUDIT_READ
    volumes:
      - certs:$CERTS_DIR:ro
      - ./metricbeat/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    depends_on:
      - elk
    environment:
      - ELASTICSEARCH_URL=["$ELASTICSEARCH_HOST"]
      - ELASTICSEARCH_USERNAME=$ELASTICSEARCH_USERNAME
      - ELASTICSEARCH_PASSWORD=$ELASTICSEARCH_PASSWORD
      - KIBANA_URL=$KIBANA_HOST