Cloudwatch logs to Kafka using Elastic agent

Elastic Stack Elastic Agent
1

Hi All,

We are trying to get Cloudwatch logs to Kafka using the elastic agent using the below config:

agent:
  logging:
    files:
      keepfiles: 7
      name: elastic-agent
      path: /var/log/elastic-agent/
      permissions: 420
    level: info
    to_files: true
inputs:
  - type: aws-cloudwatch
    region_name: "eu-west-1"
    log_group_name: "/aws/apprunner/app-admin-dev/*"
    role_arn: "arn:aws:iam::41:role/app-launcher"
monitoring:
  enabled: true
  logs: true
  metrics: true
  use_output: default
outputs:
  default:
    hosts:
    - b-1.:9094
    - b-2.:9094
    - b-3.:9094
    producer:
      compression: gzip
    ssl:
      enabled: true
      truststore_location: /etc/pki/tls/certs/kafka.client.truststore.jks
      truststore_password: ****
    topic: app_topic
    type: kafka

We are able to get the metric logs in the Kafka but the Cloudwatch logs are not coming. We could see the below error coming in the elastic-agent log files.

message":"add_cloud_metadata: received error failed with http status code 404"

Could anyone please help on getting these logs to Kafka ?