How to add AWS integration via Kibana CR?

Denis_Pudov · April 30, 2024, 3:50pm

Hello.

I tried to configure Kibana custom resource with xpack.fleet.agentPolicies to add aws integration without success. I have read many documentation and done a lot of experiments and looks that there is no support for that. Is that true?

Below are the details of some of my attempts.

The attempt to use `policy_template`:

This field is used when adding integration via the UI:

...
          package_policies:
            - name: aws
              package:
                name: aws
              namespace: quickstart
              inputs:
                - type: httpjson
                  enabled: true
                  policy_template: securityhub
...

There is error:

2024-04-30 18:13:35.386	
 FATAL  Error: [config validation of [xpack.fleet].agentPolicies.1.package_policies.0.inputs.0.policy_template]: definition for this key is missing

The attempt to disable some integrations:

...
          package_policies:
            - name: aws
              package:
                name: aws
              namespace: quickstart
              inputs:
                - type: httpjson
                  enabled: false
                - type: aws-s3
                  enabled: false
...

There are errors:

2024-04-30 18:39:16.823	
inputs.guardduty-httpjson.streams.aws.guardduty.vars.aws_region: AWS Region is required
2024-04-30 18:39:16.823	
inputs.guardduty-httpjson.streams.aws.guardduty.vars.detector_id: Detector ID is required
2024-04-30 18:39:16.823	
inputs.inspector-httpjson.streams.aws.inspector.vars.aws_region: AWS Region is required
2024-04-30 18:39:16.823	
inputs.securityhub-httpjson.streams.aws.securityhub_insights.vars.aws_region: AWS Region is required
2024-04-30 18:39:16.823	
inputs.securityhub-httpjson.streams.aws.securityhub_findings.vars.aws_region: AWS Region is required
2024-04-30 18:39:16.823	
inputs.cloudfront-aws-s3.streams.aws.cloudfront_logs.vars.queue_url: Queue URL is required
2024-04-30 18:39:16.823	
inputs.route53-aws-s3.streams.aws.route53_resolver_logs.vars.queue_url: Queue URL is required
2024-04-30 18:39:16.823	
inputs.waf-aws-s3.streams.aws.waf.vars.queue_url: Queue URL is required
2024-04-30 18:39:16.823	
inputs.s3-aws-s3.streams.aws.s3access.vars.queue_url: Queue URL is required
2024-04-30 18:39:16.823	
inputs.firewall-aws-s3.streams.aws.firewall_logs.vars.queue_url: Queue URL is required
2024-04-30 18:39:16.823	
[2024-04-30T15:39:16.822+00:00][ERROR][plugins.fleet] Package policy is invalid: inputs.elb-aws-s3.streams.aws.elb_logs.vars.queue_url: Queue URL is required

Julia_Bardi · May 21, 2024, 7:31am

Hi, you could check the created config if you add the AWS Security Hub integration from Kibana UI:

inputs:
  - id: httpjson-securityhub-0caa8e69-5656-4824-bdf5-c8716d0c880c
    name: aws-1
    revision: 1
    type: httpjson
    use_output: default
    meta:
      package:
        name: aws
        version: 2.15.2
    data_stream:
      namespace: default
    package_policy_id: 0caa8e69-5656-4824-bdf5-c8716d0c880c
    streams:
      - id: httpjson-aws.securityhub_findings-0caa8e69-5656-4824-bdf5-c8716d0c880c
        data_stream:
          dataset: aws.securityhub_findings
          type: logs
        config_version: 2
        interval: 1h
        request.timeout: 2m
        request.method: POST
        request.ssl: null
        request.url: 'https://securityhub.test.amazonaws.com/findings'
        request.transforms:
          - set:
              target: header.X-Amz-Date
              value: '[[formatDate (now) "20060102T150405Z"]]'
          - set:
              target: body.MaxResults
              value: 100
              value_type: int
          - set:
              target: body.SortCriteria
              value: '[{"Field":"UpdatedAt","SortOrder":"asc"}]'
              value_type: json
          - set:
              target: body.Filters.UpdatedAt
              value: >-
                [{ "Start": "[[formatDate (parseDate
                .cursor.last_execution_datetime "RFC3339") "2006-01-02T15"]]",
                "End": "[[formatDate (now) "2006-01-02T15"]]" }]
              default: >-
                [{ "Start": "[[formatDate (now (parseDuration "-24h"))
                "2006-01-02T15"]]", "End": "[[formatDate (now)
                "2006-01-02T15"]]" }]
              value_type: json
          - set:
              target: header.Authorization
              value: >-
                [[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256
                Credential=/%s/test/securityhub/aws4_request,
                SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now)
                "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode
                (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac
                "sha256" "AWS4" (formatDate ($now) "20060102"))) "test"))
                "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n"
                (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n"
                (formatDate ($now) "20060102") "test/securityhub/aws4_request")
                (hash "sha256" "POST\n" "/findings\n" "\n"
                "host:securityhub.test.amazonaws.com\n" (sprintf
                "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z"))
                "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]
        response.pagination:
          - set:
              target: body.NextToken
              value: >-
                [[if (eq (len .last_response.body.Findings)
                100)]][[.last_response.body.NextToken]][[end]]
              fail_on_template_error: true
          - delete:
              target: header.Authorization
          - set:
              target: header.Authorization
              value: >-
                [[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256
                Credential=/%s/test/securityhub/aws4_request,
                SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now)
                "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode
                (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac
                "sha256" "AWS4" (formatDate ($now) "20060102"))) "test"))
                "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n"
                (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n"
                (formatDate ($now) "20060102") "test/securityhub/aws4_request")
                (hash "sha256" "POST\n" "/findings\n" "\n"
                "host:securityhub.test.amazonaws.com\n" (sprintf
                "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z"))
                "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]
        cursor:
          last_execution_datetime:
            value: >-
              [[if (ne (len .last_response.body.Findings)
              100)]][[.last_event.UpdatedAt]][[end]]
        response.split:
          target: body.Findings
          ignore_empty_value: true
        tags:
          - forwarded
          - aws_securityhub_findings
        publisher_pipeline.disable_host: true
      - id: httpjson-aws.securityhub_insights-0caa8e69-5656-4824-bdf5-c8716d0c880c
        data_stream:
          dataset: aws.securityhub_insights
          type: logs
        config_version: 2
        interval: 1m
        request.timeout: 2m
        request.method: POST
        request.ssl: null
        request.url: 'https://securityhub.test.amazonaws.com/insights/get'
        request.transforms:
          - set:
              target: header.X-Amz-Date
              value: '[[(formatDate (now) "20060102T150405Z")]]'
          - set:
              target: body.MaxResults
              value: 100
              value_type: int
          - set:
              target: header.Authorization
              value: >-
                [[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256
                Credential=/%s/test/securityhub/aws4_request,
                SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now)
                "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode
                (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac
                "sha256" "AWS4" (formatDate ($now) "20060102"))) "test"))
                "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n"
                (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n"
                (formatDate ($now) "20060102") "test/securityhub/aws4_request")
                (hash "sha256" "POST\n" "/insights/get\n" "\n"
                "host:securityhub.test.amazonaws.com\n" (sprintf
                "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z"))
                "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]
        response.pagination:
          - set:
              target: body.NextToken
              value: >-
                [[if (eq (len .last_response.body.Insights)
                100)]][[.last_response.body.NextToken]][[end]]
              fail_on_template_error: true
          - delete:
              target: header.Authorization
          - set:
              target: header.Authorization
              value: >-
                [[$now := (now)]][[(sprintf "AWS4-HMAC-SHA256
                Credential=/%s/test/securityhub/aws4_request,
                SignedHeaders=host;x-amz-date, Signature=%s" (formatDate ($now)
                "20060102") (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode
                (hmac "sha256" (hexDecode (hmac "sha256" (hexDecode (hmac
                "sha256" "AWS4" (formatDate ($now) "20060102"))) "test"))
                "securityhub")) "aws4_request")) "AWS4-HMAC-SHA256\n"
                (formatDate ($now) "20060102T150405Z") "\n" (sprintf "%s/%s\n"
                (formatDate ($now) "20060102") "test/securityhub/aws4_request")
                (hash "sha256" "POST\n" "/insights/get\n" "\n"
                "host:securityhub.test.amazonaws.com\n" (sprintf
                "x-amz-date:%s\n\n" (formatDate ($now) "20060102T150405Z"))
                "host;x-amz-date\n" (hash "sha256" (sprintf `%s` .body)))))]]
        response.split:
          target: body.Insights
          ignore_empty_value: true
        tags:
          - forwarded
          - aws_securityhub_insights
        publisher_pipeline.disable_host: true

Denis_Pudov · June 7, 2024, 12:29pm

Hi, Julia!

Thank you for your reply.

It looks as low-level details that can be hidden from the user's eyes. Can we hope for a more user-friendly configuration if we consider this issue in the context of ECK?

orion-ua · September 15, 2025, 11:50am

I was trying to set up aws, aws_logs, and mongodb_atlas integrations thru xpack.fleet.agentPolicies, but have no success with any of them.

There is a lack of documentation, and error messages are often not descriptive enough.

Also, when I remove integration from xpack.fleet.agentPolicies, it may stay available in the agent policy in Kibana, and I couldn’t remove it manually as well.

Topic		Replies	Views
Fleets AWS Integration Clarification Beats fleet	5	579	June 1, 2021
Kibana with X-pack Basic connected to AWS Elasticsearch (without X-pack) Kibana	3	4423	November 2, 2018
Fleet AWS integration - looking for more info howto setup (IAM role preferred) Beats fleet	5	2100	April 26, 2021
Kibana on AWS EC2 instance to talk to Managed AWS ES Kibana	6	1654	October 5, 2017
AWS Elasticsearch - xpack_main@6.3.1 Authentication Exception from Kibana on prem Kibana	5	3287	November 15, 2018

How to add AWS integration via Kibana CR?

The attempt to use policy_template:

The attempt to disable some integrations:

Related topics

The attempt to use `policy_template`: