Filebeat stop collecting the container logs while the service keeps running!

Elastic Stack Beats
1

Hi team,

Filebeat stop collecting the container logs while the service keeps running !! When i restart the service it again start collecting .
PFA filebeat.yml

filebeat.prospectors:

Each - is a prospector. Most options can be set at the prospector level, so

you can use different prospectors for various configurations.

Below are the prospector specific configurations.

  • type: log

    Change to true to enable this prospector configuration.

    enabled: true

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • '/var/lib/docker/containers//.log.*'
      json.message_key: log
      json.keys_under_root: true
      processors:
    • add_docker_metadata: ~
      fields_under_root: true
      fields:
      type: msa_log
      #N- /home/tomcat/tomcatFCSKY/logs/catalina.out
      #N- /home/tomcat/builds/FCSKY/logs//application.log
      #N- /home/tomcat/builds/FCSKY/logs/      .log

    Exclude lines. A list of regular expressions to match. It drops the lines that are

    matching any regular expression from the list.

    #exclude_lines: ['^DBG']

    Include lines. A list of regular expressions to match. It exports the lines that are

    matching any regular expression from the list.

    #include_lines: ['^ERR', '^WARN']

    Exclude files. A list of regular expressions to match. Filebeat drops the files that

    are matching any regular expression from the list. By default, no files are dropped.

    #exclude_files: ['.gz$']

    Optional additional fields. These fields can be freely picked

    to add additional information to the crawled log files for filtering

    #fields:

    level: debug

    review: 1

    Multiline options

    Mutiline can be used for log messages spanning multiple lines. This is common

    for Java Stack Traces or C-Line Continuation

    The regexp Pattern that has to be matched. The example pattern matches all lines starting with [

    multiline.pattern: '^(([0-9]{4}-[0-9]{2}-[0-9]{2})|([a-zA-z]{3} [0-9]{2}, [0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2} [AM|PM])|([0-9]{2}-[a-zA-z]{3}-[0-9]{4})|([a-zA-z]{3} [a-zA-z]{3} [0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2})|([0-9]{2}-[0-9]{2}-[0-9]{4} [0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3})|([[0-9]{2}/[a-zA-z]{3}/[0-9]{4}))'
    #multiline.pattern: '^[[:space:]]+(at|.{3})\b|Caused by:'

    Defines if the pattern set under pattern should be negated or not. Default is false.

    multiline.negate: true

    Match can be set to "after" or "before". It is used to define if lines should be append to a pattern

    that was (not) matched before or after or as long as a pattern is not matched based on negate.

    Note: After is the equivalent to previous and before is the equivalent to to next in Logstash

    multiline.match: after

Please help as i'm loosing my log data.

Thanks,
Suresh

2

Are there any errors or warnings in the Filebeat log at the moment it stops collecting the data?

Shaunak

3

There is nothing found in logs, it mostly happened when the container switch takes place.

4

As per logs harvestor does not harvest any file when i restart the it again start harvesting the container logs

5

These are the filebeat logs running on the host.

e4a50179f60e3962ca21cf053f90d4f60801789be2f0155b3337fc1715781c-json.log
2019-10-23T08:49:38.310Z DEBUG [prospector] log/prospector.go:168 Prospector states cleaned up. Before: 4, After: 4, Pending: 0
2019-10-23T08:49:42.555Z DEBUG [docker] docker/watcher.go:207 Got a new docker event: {exec_create: /bin/sh -c curl --fail -s http://localhost:8090/manage-social/actuator/health || exit 1 3080192e485362f127d0c783aed4e8ea4356d75e03679f53110de9d61544dfa6 524350357552.dkr.ecr.us-east-1.amazonaws.com/msa-usa/fcsky-social-integration:97 container exec_create: /bin/sh -c curl --fail -s http://localhost:8090/manage-social/actuator/health || exit 1 {3080192e485362f127d0c783aed4e8ea4356d75e03679f53110de9d61544dfa6 map[com.amazonaws.ecs.container-name:fcsky-social-integration-usa com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:524350357552:task/7e6f527c-a846-415b-abb5-93e101321497 com.amazonaws.ecs.task-definition-family:fcsky-social-integration-usa com.amazonaws.ecs.task-definition-version:10 execID:f2c3ad4cccd92740e0e256e91786877f2cede06502c896456e914f5ef580d140 image:524350357552.dkr.ecr.us-east-1.amazonaws.com/msa-usa/fcsky-social-integration:97 name:ecs-fcsky-social-integration-usa-10-fcsky-social-integration-usa-a0f8caa58eabc9a02d00 com.amazonaws.ecs.cluster:ECS-USA]} local 1571820582 1571820582555482825}
2019-10-23T08:49:42.555Z DEBUG [docker] docker/watcher.go:207 Got a new docker event: {exec_start: /bin/sh -c curl --fail -s http://localhost:8090/manage-social/actuator/health || exit 1 3080192e485362f127d0c783aed4e8ea4356d75e03679f53110de9d61544dfa6 524350357552.dkr.ecr.us-east-1.amazonaws.com/msa-usa/fcsky-social-integration:97 container exec_start: /bin/sh -c curl --fail -s http://localhost:8090/manage-social/actuator/health || exit 1 {3080192e485362f127d0c783aed4e8ea4356d75e03679f53110de9d61544dfa6 map[com.amazonaws.ecs.task-definition-family:fcsky-social-integration-usa com.amazonaws.ecs.task-definition-version:10 execID:f2c3ad4cccd92740e0e256e91786877f2cede06502c896456e914f5ef580d140 image:524350357552.dkr.ecr.us-east-1.amazonaws.com/msa-usa/fcsky-social-integration:97 name:ecs-fcsky-social-integration-usa-10-fcsky-social-integration-usa-a0f8caa58eabc9a02d00 com.amazonaws.ecs.cluster:ECS-USA com.amazonaws.ecs.container-name:fcsky-social-integration-usa com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:524350357552:task/7e6f527c-a846-415b-abb5-93e101321497]} local 1571820582 1571820582555635691}
2019-10-23T08:49:42.579Z DEBUG [docker] docker/watcher.go:207 Got a new docker

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.