# Filebeat AWS CloudWatch input loss data

**URL:** <https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-loss-data/328409>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 24, 2023, 2:14am UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-loss-data/328409 "2023-03-24T02:14:07Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jacksparrow414](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jacksparrow414/32/111161_2.png) [@jacksparrow414](https://discuss.elastic.co/u/jacksparrow414)\
**Post date:** [March 24, 2023, 2:14am UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-loss-data/328409/1 "2023-03-24T02:14:07Z")

</div>

filebeat configuration

```yaml
filebeat.inputs:
  - type: aws-cloudwatch
    enabled: true
    log_group_arn: arn
    log_stream_prefix: my-logstream-prefix
    scan_frequency: 10s
    start_position: end
    access_key_id: omitting
    secret_access_key: omitting
processors:
  - drop_event:
      when:
        not:
          regexp:
            message: '^{.*$'
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
      expand_keys: true
  - rename:
      when:
        has_fields: ['reqId']
      fields:
        - from: "reqId"
          to: "trace.id"
  - rename:
      fields:
        - from: "contextPathName"
          to: "context_path_name"
  - add_tags:
      when:
        has_fields: ['error.message','error.type','error.stack_trace']
      tags: ['has error']
  - drop_fields:
      fields: ["agent", "host", "reqId","contextPathName","log.file","log.offset","input","awscloudwatch","cloud","event","log.file.path"]
      ignore_missing: true
  - drop_event:
      when:
        contains:
          context_path_name: "$"
setup.template.enabled: false
setup.template.name: "catalina-logs"
setup.template.pattern: "catalina-*"
setup.ilm.enabled: false
queue.mem:
  events: 12288
  flush.min_events: 4096
  flush.timeout: 1s
filebeat.registry.flush: 30s
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: elastic
  password: ${ELASTIC_PASSWORD}
  index: "catalina-%{[context_path_name]}"
  worker: 3
  bulk_max_size: 4096
  compression_level: 3
http.enabled: true
http.port: 15067
monitoring.enabled: false
http.host: filebeat-catalina
logging.metrics.enabled: false

```

BTW,I used to think it was a problem with the processor configuration. When I commented out all the processors, it still lost data.

lost data is not one or two, according to my observation, it is a batch of data.

I also once suspected that Elasticsearch lost the data, but after I changed the log level of filebeat to debug, I found that there was no lost data in the log about the publish event. So it's not that elasticsearch lost the data.

Now I don't know how to verify whether it pulls the log or not

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2023, 4:14am UTC](https://discuss.elastic.co/t/filebeat-aws-cloudwatch-input-loss-data/328409/2 "2023-04-21T04:14:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.