Fleet AWS Billing integration

Elastic Stack Beats
, ,
1

Hello all,

I'm trying to collect billing data with the AWS Billing integration using the elastic agent managed by Fleet. Running version 8.3.3

The agent appears to be running fine, with valid secret access key and permissions, but no data gets ingested.
The logs from the agent show the following error:

16:27:44.333
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] costexplorer GetCostAndUsageRequest failed: SerializationError: failed decoding JSON RPC error response
caused by: invalid character '<' looking for beginning of value

Running out of ideas. Should I try with metricbeat perhaps?
Any help would be greatly appreciated.

2

@Andrea_Spacca could you shed some light here? It seems to be an issue with the AWS integration.

3

Tried again using metricbeat version 7.17.0 with the following configuration:

logging.level: info
logging.selectors: ["*"]

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.kibana:
  host: "http://kibana:5601"
  username: "elastic"
  password: "<PASSWORD>"

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: ["http://elasticsearch:9200"]
  username: "elastic"
  password: "<PASSWORD>"

metricbeat.modules:
- module: aws
  period: 60s
  access_key_id: <KEY ID>
  secret_access_key: <SECRET ACCESS KEY>
  metricsets:
    - billing
  cost_explorer_config:
    group_by_dimension_keys:
      - "AZ"
      - "INSTANCE_TYPE"
      - "SERVICE"
      - "LINKED_ACCOUNT"
    group_by_tag_keys:
      - "aws:createdBy"

and I'm getting an error message:

ERROR   [aws.billing]   billing/billing.go:254  costexplorer GetCostAndUsageRequest failed: ValidationException: Start date (and hour) should be before end date (and hour)
        status code: 400, request id: 5aa02481-0576-409c-8eaf-809c4657e383

Same error message running version 8.3.3

4

Confirmed. I get the same with the elastic agent version 8.3.3.

[elastic_agent.metricbeat][error] aws GetMetricDataResults failed with error GetMetricData with Paginator: ValidationError: The parameter StartTime must not equal parameter EndTime.

	status code: 400, request id: 43614125-ec4e-46ad-8b21-6b282bc4dac7, skipping region us-east-1

[elastic_agent.metricbeat][error] costexplorer GetCostAndUsageRequest failed: SerializationError: failed decoding JSON RPC error response
caused by: invalid character '<' looking for beginning of value
5

hello @aricau

sorry for the late feedback, we'll look into that

what version of the AWS integration package are you using?

6

hello @aricau

it seems the problem reside in the period setting.
did you use a custom value by chance?

7

Hi @Andrea_Spacca . I'm using AWS Billing version 1.18.2
I did set my period to 2 hours. Is that too low?

8

hi @aricau

there is bug that provokes that error if the period is less than 24h

as a mitigation you can set that value, until the bug will be fixed

9

Had a look at the source code over in beats/billing.go at main · elastic/beats · GitHub and realised that the module is requesting daily billing data and the start & end date is calculated based on the period.

Can confirm it's now working if period is set to 24h.

Thanks.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.