Filebeat and ECS

Elastic Stack Beats
,
1

Hey guys,

We currently have filebeat running parallel with other ecs tasks in our ecs cluster. Each filebeat logs data from /var/lib/dockers/containers//.log and sends them to our elasticsearch and ultimately shows up in our Kibana. All the data is there, however from the fields, 'container.name' in particular only ever shows "ecs-agent" and not the name of the container of the actual task running in that instance. Has anyone else got this issue or have a work-around to get the correct container/image names?

This is what out filebeat.yml looks like:

filebeat.inputs:
- type: container
  paths: 
    - '/var/lib/docker/containers/*/*.log'
    - '/var/log/ecs/ecs-agent.log.*'

setup.template.name: "filebeat-${ENVIRONMENT:default}"
setup.template.pattern: "${ENVIRONMENT:default}-ecs-%{+yyyy.MM.dd}-*"
setup.template.overwrite: true
setup.ilm.enabled: false
ilm.enabled: false

output.elasticsearch:
  hosts: ["https://<elasticsearch-url>:443"]
  index: "${ENVIRONMENT:default}-ecs-%{+yyyy.MM.dd}"

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

Thanks,
Eddy

2

Hey @edmond.qiu!

What version of Filebeat do you use?

3

Hi @ChrsMark
The version we're using is 7.6.2

4

Hi again @edmond.qiu, I was not able to reproduce it with the latest master of the upstream project. Could you provide a more specific configuration and environment information in order to try to reproduce it?

5

We deploy the ecs service with this task definition:
[
{
"name": "filebeat-daemon",
"image": "(private-ecs-image-of-service)",
"cpu": 128,
"memory": 128,
"essential": true,
"environment": [{
"name": "ENVIRONMENT",
"value": "{environment}" }], "mountPoints": [ { "sourceVolume": "containers", "containerPath": "/var/lib/docker/containers", "readOnly": true }, { "sourceVolume": "socket", "containerPath": "/var/run/docker.sock", "readOnly": true } ], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/aws/ecs/filebeat/{environment}",
"awslogs-region": "{region}", "awslogs-stream-prefix": "{environment}"
}
}
}
]

and the dockerfile of the private image pulled:

FROM docker.elastic.co/beats/filebeat:7.6.2

USER root

COPY filebeat.docker.yml /usr/share/filebeat/filebeat.yml

CMD ["filebeat", "-e", "-strict.perms=false"]

Also, is there any way for us to review the test setup on your end, to see what kind of field and values you are getting?

6

Hi!

The config I tested on my local env is this:

filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'


filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

output.console:
  pretty: true

processors:
  - add_docker_metadata: ~

Then I start Filebeat like:
sudo -E ./filebeat --strict.perms=false | jq '.container.name'
and I see proper names or null (when container name is not added) but nothing similar to what you mention.

7

Yeah, when I test on my local env it works perfectly showing the correct container name and all. But when it comes from the AWS ecs service it starts displaying the odd behaviour.

8

@edmond.qiu It might happen that this is how metadata are provided from this service. Does it happen to know from where ecs-agent comes from? It might worth opening a Github issue for this for further investigation.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.