Aws ecs filebeat log collection stops when the application containers are restarted

Elastic Stack Beats
,
1

We collect the logs of the container of our java application on ecs with the container of Filebeat.
When any one or multiple container in the AWS ECS intance restart continously, filebeat container stops pushing logs to elasticsearch.

And if we restart the filebeat container then the logs starts coming.

When I checked the Filebeat log, there were no errors.

filebeat config file (filebeat.yml)

setup.ilm.enabled: false
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      json.message_key: log
      templates:
        - condition:
            equals:
              docker.container.labels.filebeat_enable: "true"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              json.keys_under_root: true
              json.add_error_key: false
              multiline.pattern: '^\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.*'
              multiline.negate: true
              multiline.match: after
              multiline.timeout: 120s
              multiline.max_lines: 1000
processors:
- add_cloud_metadata: ~
- add_docker_metadata: ~
output.elasticsearch:
  hosts: ["http://x.x.x.x:9200","http://x.x.x.x:9200","http://x.x.x.x:9200"]
  username: "XXX"
  password: "XXXX"
2

@system please help me out here

3

@Tetiana_Kravchenko Could you take a look at this one please? TIA!!

4

Hi @vaseem_tt

I think that you need to use container as the config.type.
Try to use this configuration (below the co.elastic.logs/pipeline) with docker as a autodiscover provider and container as a template.config.type.
Also if you are not using hints based autodiscovery - hints.enabled can be removed from the configuration.

5

@Tetiana_Kravchenko
we have updated the filebeat configuration as like you have mentioned above. But still the issue persists.

setup.ilm.enabled: false
filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      json.message_key: log
      templates:
        - condition:
            equals:
              docker.container.labels.filebeat_enable: "true"
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              max_bytes: 1048576
              json.keys_under_root: true
              json.add_error_key: false
              multiline.pattern: '^\[[0-9]{4}-[0-9]{2}-[0-9]{2}'
              multiline.negate: true
              multiline.match: after
              multiline.timeout: 120s
              multiline.max_lines: 1000
              close_inactive: 30s
              scan_frequency: 10s
6

@system please can someone check this?

7

@system please look into this

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.