# Monitor specific AWS Cloudwatch log\_stream

**URL:** <https://discuss.elastic.co/t/monitor-specific-aws-cloudwatch-log-stream/315395>\
**Category:** Elastic Agent\
**Tags:** fleet, integrations\
**Created:** [September 28, 2022, 5:35pm UTC](https://discuss.elastic.co/t/monitor-specific-aws-cloudwatch-log-stream/315395 "2022-09-28T17:35:19Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\
**Post date:** [September 28, 2022, 5:35pm UTC](https://discuss.elastic.co/t/monitor-specific-aws-cloudwatch-log-stream/315395/1 "2022-09-28T17:35:19Z")

</div>

I'm using a Fleet-managed Elastic Agent with the AWS Custom Log integration to monitor logs from Cloudwatch. My configuration in Fleet generates the following in the policy:

```yaml
- id: aws-cloudwatch-aws_logs-334607ab-b4fe-4ec7-953b-c34b2bdbcbd4
  name: my_app.log
  revision: 14
  type: aws-cloudwatch
  use_output: default
  meta:
    package:
      name: aws_logs
      version: 0.2.3
  data_stream:
    namespace: default
  streams:
    - id: aws-cloudwatch-aws_logs.generic-334607ab-b4fe-4ec7-953b-c34b2bdbcbd4
      data_stream:
        dataset: my_app.log
      start_position: end
      log_group_name: /my_app/logs
      api_sleep: 200ms
      endpoint: amazonaws.com
      log_streams: 'app/my_app/log,batch/my_app/log'
      region_name: us-east-1
      processors:
        - add_fields:
            when.not.has_fields:
              - service.environment
            fields:
              environment: '${env.ENV}'
            target: service
        - add_locale:
            format: offset
      scan_frequency: 1m
      tags:
        - preserve_original_event
        - forwarded
        - my_app
      publisher_pipeline.disable_host: true

```

However, I get the following error from `elastic_agent.filebeat`:

```auto
getLogEventsFromCloudWatch failed: error FilterLogEvents with Paginator: operation error CloudWatch Logs: FilterLogEvents, https response error StatusCode: 400, RequestID: 5d24e028-cbe3-4e0d-a261-62c40c087cc0, ResourceNotFoundException: The specified log stream does not exist.

```

I get the same error if I only include one `log_stream` in my fleet config. If I remove the log streams, then logs ingest as expected, and I can see in the `aws.cloudwatch.log_stream` field that these streams do exist.

The issue with doing this is that a number of the streams being sent to `/my_app/logs` require different pipelines and a few should go to different datasets, which is why this is ideal.

I've tested this with the AWS Cloudwatch logs integration with the same result. The only thing that I can see is that [the documentation](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-cloudwatch.html#_log_streams) defines `log_streams` as:

> **`log_streams`**
> 
> A list of strings of log stream names that Filebeat collects log events from

It doesn't mention a comma-separated string, but I know that many settings seem to allow for either/or.

Is this something I'm doing incorrectly, or is this a bug that needs to be reported?

Thx.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2022, 5:35pm UTC](https://discuss.elastic.co/t/monitor-specific-aws-cloudwatch-log-stream/315395/2 "2022-10-26T17:35:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.