# Filebeats 7.4.2/nomad OOM error

**URL:** <https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 11, 2019, 6:13pm UTC](https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407 "2019-11-11T18:13:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![djesrani](https://avatars.discourse-cdn.com/v4/letter/d/bbe5ce/32.png) [@djesrani](https://discuss.elastic.co/u/djesrani)\
**Post date:** [November 11, 2019, 6:13pm UTC](https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407/1 "2019-11-11T18:13:39Z")

</div>

Hello all,

I've been running into Out Of Memory/"task killed" issues deploying Filebeats and docker-gen into our production cluster.

Nomad version is 0.9.4, Filebeats version is 7.4.2. Memory allocation to Filebeats by Nomad is currently set to 256 MB, logs - max\_files is set to 10, max\_file\_size is set to 15.

I've been reading that there were memory leak issues associated with this type of problem, but they seem to have been remediated since 6.x and subsequently closed.

Filebeats 7.4.2 was deployed for about a week before these problems started to re-appear so that seems to speak to something related to accumulation/lack of GC/harvesting?

Am I missing some sort of configuration option, environment variable or flag which might mitigate this kind of thing?

Any insight would be greatly appreciated.

Thanks very much in advance,

Darshan Jesrani  
Analytic Partners, Inc.  
New York , NY

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [November 13, 2019, 2:39pm UTC](https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407/2 "2019-11-13T14:39:17Z")

</div>

Hi @djesrani,

Could you share some more details about your deployment? Including the config file.

Best regards

---

<div class="post-metadata">

**Author:** ![djesrani](https://avatars.discourse-cdn.com/v4/letter/d/bbe5ce/32.png) [@djesrani](https://discuss.elastic.co/u/djesrani)\
**Post date:** [November 13, 2019, 5:15pm UTC](https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407/3 "2019-11-13T17:15:13Z")

</div>

Hi @exekias , thanks for asking -- below is the config:

job "filebeats" {  
type = "system"  
datacenters = ["[[.datacenter.value]]"]

group "filebeat" {  
task "filebeat" {  
driver = "docker"  
user = "root"

```
  config {
    image = "docker.elastic.co/beats/filebeat:7.4.2"

    volumes = [
      "/var/run/docker.sock:/var/run/docker.sock",
      "/var/lib/docker:/var/lib/docker",
      "/var/nomad/alloc:/var/nomad/alloc",
      "local/filebeat_yml:/usr/share/filebeat/filebeat.yml",
    ]

    dns_servers = ["${attr.unique.network.ip-address}"]
  } #conf

  env {
    ELASTICSEARCH_HOSTS = "elk.service.lg1.consul:9200"
    ELASTICSEARCH_USERNAME = "my_login"
    ELASTICSEARCH_PASSWORD = "my_password"
  }

  resources {
    memory = 256
  } #res

  logs {
    max_files = 10
    max_file_size = 15
  }

  template {
    data = <<EOH

```

filebeat.config:  
modules:  
path: ${path.config}/modules.d/\*.yml  
reload.enabled: false

processors:

- add\_cloud\_metadata: ~

output.elasticsearch:  
hosts: '{ELASTICSEARCH\_HOSTS}' username: '{ELASTICSEARCH\_USERNAME}'  
password: '${ELASTICSEARCH\_PASSWORD}'

filebeat.config.inputs:  
enabled: true  
path: /alloc/data/\*.yml  
reload.enabled: true  
reload.period: 10s  
EOH

```
    destination = "local/filebeat_yml"
  }
}

task "dockergen" {
  leader = true
  driver = "docker"

  config {
    image = "jwilder/docker-gen:latest"

    args = [
      "-watch",
      "-include-stopped",
      "-wait",
      "5s:10s",
      "/etc/docker-gen/templates/filebeat.tmpl",
      "/alloc/data/inputs.yml",
    ]

    volumes = [
      "/var/run/docker.sock:/tmp/docker.sock",
      "local/filebeat_tmpl:/etc/docker-gen/templates/filebeat.tmpl",
    ]

    dns_servers = ["${attr.unique.network.ip-address}"]
  }

  env {
    "NOMAD_ALLOC" = "/var/nomad/alloc"
    "SERVER_HOST" = "${node.unique.name}"
  }

  template {
    left_delimiter = "{{{"
    right_delimiter = "}}}"

    data = <<EOH

```

{{/\* this is a docker-gen template. See docker-gen for more details _/}}  
{{ $nomad\_alloc := .Env.NOMAD\_ALLOC }}  
{{ range $key, $value := . }}  
{{ $nomad\_id := $value.Env.NOMAD\_ALLOC\_ID }}  
- type: log  
paths:  
{{ if $nomad\_id }}  
- {{ $nomad\_alloc }}/{{ $nomad\_id }}/alloc/logs/{{ $value.Env.NOMAD\_TASK\_NAME }}.stdout._  
- {{ $nomad\_alloc }}/{{ $nomad\_id }}/alloc/logs/{{ $value.Env.NOMAD\_TASK\_NAME }}.stderr.\*  
{{ else }}  
- /var/lib/docker/containers/{{ $value.ID }}/{{ $value.ID }}-json.log  
{{ end }}  
document\_type: docker  
ignore\_older: 30m  
scan\_frequency: 5s  
fields\_under\_root: true  
fields:  
docker/id: {{ $value.ID }}  
docker/image: {{ if $value.Image.Registry }}{{ $value.Image.Registry }}/{{ end }}{{ $value.Image.Repository }}{{ if $value.Image.Tag }}:{{ $value.Image.Tag }}{{ else }}:latest{{ end }}  
docker/name: {{ $value.Name }}  
{{ if $nomad\_id }}docker/task\_name: {{ $value.Env.NOMAD\_TASK\_NAME }}{{end}}  
{{ if $nomad\_id }}docker/nomad\_dc: {{ $value.Env.NOMAD\_DC }}{{end}}  
{{ if $nomad\_id }}docker/nomad\_alloc\_id: {{ $value.Env.NOMAD\_ALLOC\_ID }}{{end}}  
{{ if $nomad\_id }}docker/nomad\_alloc\_index: {{ $value.Env.NOMAD\_ALLOC\_INDEX }}{{end}}  
docker/hostname: {{ value.Hostname }} docker/server\_host: {{ .Env.SERVER\_HOST }}  
{{ range $k, $v := $value.Labels }}  
docker/label/{{ $k }}: {{ $v }}  
{{ end }}  
{{ end }}  
EOH

```
    destination = "local/filebeat_tmpl"
  }

  resources {
    memory = 128
  }

  logs {
    max_files = 10
    max_file_size = 15
  }
}

```

}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 11, 2019, 7:15pm UTC](https://discuss.elastic.co/t/filebeats-7-4-2-nomad-oom-error/207407/4 "2019-12-11T19:15:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.