Heartbeat (7.15.2) - aws_elb autodiscover only sends data for single listener while more should be included

Elastic Stack Beats
1

I try to get heartbeat with aws_elb autodiscovery to work. Though I'm able to get data visible in kibana, it is always a single listener from a single lb. Whereas I have 3 lb setup and it logs that it fetched 16 listeners.

I tried all kinds of configs with conditions without conditions etc etc. But I keep running into the scenario that autodiscovery logs only one start event.

heartbeat.autodiscover:
  providers:
  - type: aws_elb
    period: 5m
    regions: ["eu-west-1"]
    templates:
    - condition:
       equals.port: 3000
      config:
      - type: tcp
        name: "${data.host}:${data.port}"       
        hosts: ["${data.host}:${data.port}"]
        schedule: "@every 5s"
        timeout: 1s
    - condition:
       equals.port: 5601
      config:
      - type: http
        hosts: ["${data.host}:${data.port}"]
        schedule: "@every 5s"
        timeout: 1s

Whatever conditions I add or remove the only listener for which heartbeat is registering events is the last listener from the last lb when I output the data via the was cli. In my case port 3000.

If anyone knows what is going wrong in my setup, please enlighten me.

Logs showing single listener start event

Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]: 2022-01-07T15:55:12.082+0100        DEBUG        [autodiscover]        autodiscover/autodiscover.go:172        Got a start event.        {"autodiscover.event": {"aws":{"elb":{"availability_zones":["eu-west-1c","eu-west-1a","eu-west-1b"],"created":"2022-01-07T08:18:41.43Z","host":"internal-elk7-t-cg-a-stage-client-alb-306711211.eu-west-1.elb.amazonaws.com","ip_address_type":"ipv4","listener_arn":"arn:aws:elasticloadbalancing:eu-west-1:807891339983:listener/app/elk7-t-cg-a-stage-client-alb/1727709932621845/e87d83211f006b90","load_balancer_arn":"arn:aws:elasticloadbalancing:eu-west-1:807891339983:loadbalancer/app/elk7-t-cg-a-stage-client-alb/1727709932621845","port":3000,"protocol":"HTTP","scheme":"internal","security_groups":["sg-0e28a298b7f1d3a33"],"ssl_policy":null,"state":{"code":"active"},"type":"application","vpc_id":"vpc-4fac482b"}},"cloud":{"availability_zone":["eu-west-1c","eu-west-1a","eu-west-1b"],"provider":"aws","region":"eu-west-"},"config":[{}],"host":"internal-elk7-t-cg-a-stage-client-alb-306711211.eu-west-1.elb.amazonaws.com","id":"arn:aws:elasticloadbalancing:eu-west-1:807891339983:listener/app/elk7-t-cg-a-stage-client-alb/1727709932621845/e87d83211f006b90","meta":{"aws":{"elb":{"availability_zones":["eu-west-1c","eu-west-1a","eu-west-1b"],"created":"2022-01-07T08:18:41.43Z","host":"internal-elk7-t-cg-a-stage-client-alb-306711211.eu-west-1.elb.amazonaws.com","ip_address_type":"ipv4","listener_arn":"arn:aws:elasticloadbalancing:eu-west-1:807891339983:listener/app/elk7-t-cg-a-stage-client-alb/1727709932621845/e87d83211f006b90","load_balancer_arn":"arn:aws:elasticloadbalancing:eu-west-1:807891339983:loadbalancer/app/elk7-t-cg-a-stage-client-alb/1727709932621845","port":3000,"protocol":"HTTP","scheme":"internal","security_groups":["sg-0e28a298b7f1d3a33"],"ssl_policy":null,"state":{"code":"active"},"type":"application","vpc_id":"vpc-4fac482b"}},"cloud":{"availability_zone":["eu-west-1c","eu-west-1a","eu-west-1b"],"provider":"aws","region":"eu-west-"}},"port":3000,"provider":"c2f082a2-f2c9-4544-814f-26d6558f1b04","start":true}}

Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]: 2022-01-07T15:55:12.082+0100        DEBUG        [autodiscover]        autodiscover/autodiscover.go:193        Generated config: {
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   "hosts": [
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:     "xxxxx"
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   ],
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   "name": "internal-elk7-t-cg-a-stage-client-alb-306711211.eu-west-1.elb.amazonaws.com:3000",
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   "schedule": "@every 5s",
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   "timeout": "1s",
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]:   "type": "tcp"
Jan 07 15:55:12 ip-172-24-25-17 heartbeat[4167]: }
2

Terribly sorry to hear about this, that certainly should not be the case! I think you've found a bug.

It's at this point that I have to remark that you may be the only person I've heard of trying this feature in the two years since it was introduced, for that reason it's been in experimental mode since introduction, without ever being promoted to 'beta'. I had high hopes for this feature, but to be completely honest, it seems to be a bit of a dud.

Is this a bug you'd be able / interesting in contributing to our codebase to fix? The reason I ask is that it's probably not going to be prioritized as a feature to fix based on that low rate of usage.

If so, we'd be glad to review / merge a PR (the key code is here). If not, to be honest, it's probably something we're more inclined to remove than maintain given the very low usage.

I'm terribly sorry for the bad experience you've had with this feature, it's a bit of an unusual case, usually experimental features get traction and are promoted to beta then GA, but this one seems to just not be very popular.

3

Then I will implement it a bit less dynamic. I was not planning on working on this feature. Especially not when you point out that nobody uses it... Maybe it is better to let it vanish slowly from the stack.

4

Thanks for understanding, I've opened [Heartbeat] Remove AWS ELB autodiscovery · Issue #29782 · elastic/beats · GitHub to deprecate this feature.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.