Custom service monitoring and autodiscovery

[denisk (Denis)]

Hi,

I have set up metric collection of a number of custom microservices using the http module. Per microservice I've create a separate yaml file, and in each yaml file looks something like this:

# Each service has its own yaml file like below with custom setup
- module: http
  metricsets :
    - json
  period : 60s
  hosts : ["http://${MY_SERVICE1}:${SERVICE1_PORT}"]
  path: "/management/health"
  namespace : 'service1'
  fields_under_root: true
  fields :
    environment: '${ENVIRONMENT:local}'
    systemname: system-name
    servicename: service-name
    logtype : metricbeat

- module: http
  metricsets :
    - json
  period : 60s
  hosts : ["http://${MY_SERVICE1}:${SERVICE1_PORT}"]
  path: "/management/metrics/my.metric1"
  namespace : 'service1'
  fields_under_root: true
  fields :
    environment: '${ENVIRONMENT:local}'
    systemname: system-name
    servicename: service-name
    logtype : metricbeat

... etc

I customized the metricbeat container and put the yaml files in the right location:

# Metricbeat Dockerfile
FROM docker.elastic.co/beats/metricbeat:8.6.2
COPY my-metricbeat.yml /usr/share/metricbeat/metricbeat.yml
COPY modules.d/* /usr/share/metricbeat/modules.d/
...

When I enable my custome custom modules from the start, metricbeat automatically starts polling my services and everything works nicely. However that is not a solid solution for when services are not always running, or if services get replicated and I want to monitor all instances.

So I started with autodiscovery. In the metricbeat.yml file I enable autodiscovery:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 30s

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

In the docker-compose file I added these labels to the services:

    labels:
      - "co.elastic.metrics/hosts=$${data.host}:$${data.port}"
      - "co.elastic.metrics/period=60s"
      - "co.elastic.metrics/module=service1"

When I start those services, I can see that metricbeat notices the startup, but doesn't like the configuration:

{
  "log.level": "debug",
  "@timestamp": "2023-04-04T08:49:51.446Z",
  "log.logger": "autodiscover",
  "log.origin": {
    "file.name": "autodiscover/autodiscover.go",
    "file.line": 203
  },
  "message": "Generated config: {\n  \"enabled\": true,\n  \"hosts\": [\n    \"xxxxx\"\n  ],\n  \"metricsets\": [],\n  \"module\": \"my-service1\",\n  \"period\": \"60s\",\n  \"processors\": [],\n  \"timeout\": \"3s\"\n}",
  "service.name": "metricbeat",
  "ecs.version": "1.6.0"
}
{
  "log.level": "debug",
  "@timestamp": "2023-04-04T08:49:51.447Z",
  "log.logger": "autodiscover",
  "log.origin": {
    "file.name": "autodiscover/autodiscover.go",
    "file.line": 290
  },
  "message": "Got a meta field in the event",
  "service.name": "metricbeat",
  "ecs.version": "1.6.0"
}
{
  "log.level": "error",
  "@timestamp": "2023-04-04T08:49:51.447Z",
  "log.logger": "autodiscover",
  "log.origin": {
    "file.name": "autodiscover/autodiscover.go",
    "file.line": 236
  },
  "message": "Auto discover config check failed for config '{\n  \"enabled\": true,\n  \"hosts\": [\n    \"xxxxx\"\n  ],\n  \"metricsets\": [],\n  \"module\": \"service1\",\n  \"period\": \"60s\",\n  \"processors\": [],\n  \"timeout\": \"3s\"\n}', won't start runner: no metricsets configured for module 'service1'",
  "service.name": "metricbeat",
  "ecs.version": "1.6.0"
}

I was wondering: I have the idea that I'm stretching the setup of metricbeat, because I've created my own module, just by creating a new file in the modules.d/ directory, with a reused of the http module and metricset json. Can this setup with autodiscovery actually work, or should I search in a different direction?