# Filebeat and ECS

**URL:** <https://discuss.elastic.co/t/filebeat-and-ecs/231429>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [May 7, 2020, 12:07am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429 "2020-05-07T00:07:47Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![edmond.qiu](https://avatars.discourse-cdn.com/v4/letter/e/7ea924/32.png) [@edmond.qiu](https://discuss.elastic.co/u/edmond.qiu)\
**Post date:** [May 7, 2020, 12:07am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/1 "2020-05-07T00:07:47Z")

</div>

Hey guys,

We currently have filebeat running parallel with other ecs tasks in our ecs cluster. Each filebeat logs data from /var/lib/dockers/containers/_/_.log and sends them to our elasticsearch and ultimately shows up in our Kibana. All the data is there, however from the fields, 'container.name' in particular only ever shows "ecs-agent" and not the name of the container of the actual task running in that instance. Has anyone else got this issue or have a work-around to get the correct container/image names?

This is what out filebeat.yml looks like:

```auto
filebeat.inputs:
- type: container
  paths: 
    - '/var/lib/docker/containers/*/*.log'
    - '/var/log/ecs/ecs-agent.log.*'

setup.template.name: "filebeat-${ENVIRONMENT:default}"
setup.template.pattern: "${ENVIRONMENT:default}-ecs-%{+yyyy.MM.dd}-*"
setup.template.overwrite: true
setup.ilm.enabled: false
ilm.enabled: false

output.elasticsearch:
  hosts: ["https://<elasticsearch-url>:443"]
  index: "${ENVIRONMENT:default}-ecs-%{+yyyy.MM.dd}"

processors:
- add_docker_metadata:
    host: "unix:///var/run/docker.sock"

```

Thanks,  
Eddy

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [May 7, 2020, 8:00am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/2 "2020-05-07T08:00:28Z")

</div>

Hey @edmond.qiu!

What version of Filebeat do you use?

---

<div class="post-metadata">

**Author:** ![edmond.qiu](https://avatars.discourse-cdn.com/v4/letter/e/7ea924/32.png) [@edmond.qiu](https://discuss.elastic.co/u/edmond.qiu)\
**Post date:** [May 7, 2020, 10:34am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/3 "2020-05-07T10:34:22Z")

</div>

Hi @ChrsMark  
The version we're using is 7.6.2

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [May 8, 2020, 9:08am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/4 "2020-05-08T09:08:46Z")

</div>

Hi again @edmond.qiu, I was not able to reproduce it with the latest master of the upstream project. Could you provide a more specific configuration and environment information in order to try to reproduce it?

---

<div class="post-metadata">

**Author:** ![edmond.qiu](https://avatars.discourse-cdn.com/v4/letter/e/7ea924/32.png) [@edmond.qiu](https://discuss.elastic.co/u/edmond.qiu)\
**Post date:** [May 11, 2020, 12:01am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/5 "2020-05-11T00:01:02Z")

</div>

We deploy the ecs service with this task definition:  
[  
{  
"name": "filebeat-daemon",  
"image": "(private-ecs-image-of-service)",  
"cpu": 128,  
"memory": 128,  
"essential": true,  
"environment": [{  
"name": "ENVIRONMENT",  
"value": "{environment}" }], "mountPoints": [{ "sourceVolume": "containers", "containerPath": "/var/lib/docker/containers", "readOnly": true }, { "sourceVolume": "socket", "containerPath": "/var/run/docker.sock", "readOnly": true }], "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "/aws/ecs/filebeat/{environment}",  
"awslogs-region": "{region}", "awslogs-stream-prefix": "{environment}"  
}  
}  
}  
]

and the dockerfile of the private image pulled:

```auto
FROM docker.elastic.co/beats/filebeat:7.6.2

USER root

COPY filebeat.docker.yml /usr/share/filebeat/filebeat.yml

CMD ["filebeat", "-e", "-strict.perms=false"]

```

Also, is there any way for us to review the test setup on your end, to see what kind of field and values you are getting?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [May 11, 2020, 9:26am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/6 "2020-05-11T09:26:08Z")

</div>

Hi!

The config I tested on my local env is this:

```auto
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1

output.console:
  pretty: true

processors:
  - add_docker_metadata: ~

```

Then I start Filebeat like:  
`sudo -E ./filebeat --strict.perms=false | jq '.container.name'`  
and I see proper names or `null` (when container name is not added) but nothing similar to what you mention.

---

<div class="post-metadata">

**Author:** ![edmond.qiu](https://avatars.discourse-cdn.com/v4/letter/e/7ea924/32.png) [@edmond.qiu](https://discuss.elastic.co/u/edmond.qiu)\
**Post date:** [May 11, 2020, 11:31pm UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/7 "2020-05-11T23:31:47Z")

</div>

Yeah, when I test on my local env it works perfectly showing the correct container name and all. But when it comes from the AWS ecs service it starts displaying the odd behaviour.

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [May 12, 2020, 8:50am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/8 "2020-05-12T08:50:42Z")

</div>

@edmond.qiu It might happen that this is how metadata are provided from this service. Does it happen to know from where `ecs-agent` comes from? It might worth opening a Github issue for this for further investigation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 9, 2020, 8:50am UTC](https://discuss.elastic.co/t/filebeat-and-ecs/231429/9 "2020-06-09T08:50:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.