Configuring aws ec2 on metricbeat k8 deamonset throws errors

[eyesmoker1]

Running es 7.5 and metricbeat 7.5 helm charts on k8s. Trying to get ec2 dashboard working on kibana but metricbeat throws following errors.

Metricbeat code
- module: aws period: 300s metricsets: - ec2 - rds access_key_id: 'XXXXX' secret_access_key: 'XXXX' output.elasticsearch: hosts: '${XXXX:9200}'

Metricbeat pod logs:
"metricset":common.MapStr{"name":"ec2", "period":300000}, "service":common.MapStr{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.tags.kubernetes] of type [keyword] in document with id 'b1okqG8B_OfmuHL1SKga'. Preview of field's value: '{io/cluster/lakeplacid-eksCluster-0a550bd=owned}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:365"}}

Please help.

[Kaiyan_Sheng]

Hi @eyesmoker, thanks for posting your question here. This issue is caused by label/annotation dots ( . ) creating hierarchy in Elasticsearch documents. Let me create a github issue to fix this. Thank you!

[eyesmoker1]

Thank you very much. Is there a workaround for this? Is this happening only in 7.5.0 and do you recommend 7.5.1?
Please point me to the GitHub issue.
This is an important feature requirement for us to see ec2 observability.

[Kaiyan_Sheng]

Here is the ticket: https://github.com/elastic/beats/issues/15843
Unfortunately I don't see a workaround besides not using dot in kubernetes labels. Or if you don't mind collecting metrics without the label, you can use cloudwatch metricset with config:

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  metrics:
    - namespace: AWS/EC2

I do have a PR in review already to add dedot for tags: https://github.com/elastic/beats/pull/15844

[eyesmoker1]

Thank you. Will this be part of next release?

[Kaiyan_Sheng]

Yes! This fix will be in 7.6.0 release. Thanks!