Autodiscover and metadata

Elastic Stack Beats
,
1

Hi ,
I've just tried the new version of Heartbeat (6.5.4) and specifically the autodiscover feature with Docker.
For the POC , I launched a Redis and Nginx container along with heartbeat (installed like a linux service).
Both containers have been discovered by heartbeat with the right ports (80 and 6379) but the data don't record the docker metadata like container_name, ID or image name.
In kibana , I've just found these data :

indent preformatted text by 4 spaces
{
 "_index": "heartbeat-6.5.4-2019.01.04",
 "_type": "doc",
 "_id": "Or1rGWgBJnsBp6ZGGWZ3",
 "_version": 1,
 "_score": null,
 "_source": {
"@timestamp": "2019-01-04T15:11:15.719Z",
"beat": {
  "name": "XXXXXXXXXX",
  "hostname": "XXXXXXXXXXXXXX",
  "version": "6.5.4"
},
"tcp": {
  "rtt": {
    "connect": {
      "us": 234
    }
  },
  "port": 80
},
"monitor": {
  "duration": {
    "us": 353
  },
  "status": "up",
  "host": "172.17.0.3",
  "scheme": "tcp",
   "id": "tcp-tcp@172.17.0.3:80",
  "type": "tcp",
  "name": "tcp",
  "ip": "172.17.0.3"
},
"host": {
  "containerized": true,
  "architecture": "x86_64",
  "os": {
    "platform": "rhel",
    "version": "7.5 (Maipo)",
    "family": "",
    "codename": "Maipo"
  },
  "name": "XXXXXXXXXXX",
  "id": "d3dacd4457e5455e806326a7b6b08cc3"
  }
  },
  "fields": {
 "@timestamp": [
   "2019-01-04T15:11:15.719Z"
 ]
},
 "highlight": {
"monitor.host": [
  "@kibana-highlighted-field@172.17.0.3@/kibana-highlighted-field@"
]
 },
  "sort": [
   1546614675719
 ]
}

Without the docker metadata the informations are not useful and valueless.
Thank you for your help.
Eric

2

That's a great point. Luckily this is possible. I've actually just created a pull request to better document adding those fields. You can interpolate any fields you like with the standard fields option, which is per-monitor.

For convenience I've included an example below. You may also want to set fields_under_root: true in your config if you'd prefer for those fields to be at the root level of your ES documents instead of the fields key.

heartbeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - type: tcp
              hosts: ["${data.host}:${data.port}"]
              schedule: "@every 1s"
              timeout: 1s
              fields:
                container_id: "${data.docker.container.id}"
                container_image: "${data.docker.container.image}"
                container_name: "${data.docker.container.name}"
1 Like
3

Thank you Andrew for your reply.
I didn't know this feature.
I'll test it on next week.
Yes, I'll use "field_under_root" as well.
Thx again !
Eric

4

Tested with success !
I've added the label metadata as well.
It's more useful now :wink:
Thx
Eric

1 Like
5

This will be better addressed in 7.0 where we'll just add the right fields by default. You can follow this PR here: https://github.com/elastic/beats/pull/10258 Once that happens there will be nothing to configure, and things will just work, no fancy templates required :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.