Filebeat AWS CloudWatch input loss data

Elastic Stack Beats
1

filebeat configuration

filebeat.inputs:
  - type: aws-cloudwatch
    enabled: true
    log_group_arn: arn
    log_stream_prefix: my-logstream-prefix
    scan_frequency: 10s
    start_position: end
    access_key_id: omitting
    secret_access_key: omitting
processors:
  - drop_event:
      when:
        not:
          regexp:
            message: '^{.*$'
  - decode_json_fields:
      fields: ["message"]
      target: ""
      overwrite_keys: true
      expand_keys: true
  - rename:
      when:
        has_fields: ['reqId']
      fields:
        - from: "reqId"
          to: "trace.id"
  - rename:
      fields:
        - from: "contextPathName"
          to: "context_path_name"
  - add_tags:
      when:
        has_fields: ['error.message','error.type','error.stack_trace']
      tags: ['has error']
  - drop_fields:
      fields: ["agent", "host", "reqId","contextPathName","log.file","log.offset","input","awscloudwatch","cloud","event","log.file.path"]
      ignore_missing: true
  - drop_event:
      when:
        contains:
          context_path_name: "$"
setup.template.enabled: false
setup.template.name: "catalina-logs"
setup.template.pattern: "catalina-*"
setup.ilm.enabled: false
queue.mem:
  events: 12288
  flush.min_events: 4096
  flush.timeout: 1s
filebeat.registry.flush: 30s
output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: elastic
  password: ${ELASTIC_PASSWORD}
  index: "catalina-%{[context_path_name]}"
  worker: 3
  bulk_max_size: 4096
  compression_level: 3
http.enabled: true
http.port: 15067
monitoring.enabled: false
http.host: filebeat-catalina
logging.metrics.enabled: false

BTW,I used to think it was a problem with the processor configuration. When I commented out all the processors, it still lost data.

lost data is not one or two, according to my observation, it is a batch of data.

I also once suspected that Elasticsearch lost the data, but after I changed the log level of filebeat to debug, I found that there was no lost data in the log about the publish event. So it's not that elasticsearch lost the data.

Now I don't know how to verify whether it pulls the log or not

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.