# ElasticSearch nodes create a lot of logs

**URL:** <https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514>\
**Category:** Elasticsearch\
**Created:** [August 11, 2020, 8:47am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514 "2020-08-11T08:47:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\
**Post date:** [August 11, 2020, 8:47am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/1 "2020-08-11T08:47:13Z")

</div>

Hi guys,

I have seen that the elasticsearch nodes are creating a very huge amount of logs in my ECK cluster.  
Is there a way to reduce the amount of logs?

Right now I am seeing the index being flooded by entries like this one:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/6/e64e002e03155399b7024754189790243eaed38b.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 11, 2020, 9:20am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/2 "2020-08-11T09:20:26Z")

</div>

We aren't all guys 🙂

How are you pushing these logs, using the Elasticsearch module in Filebeat, or something else?

---

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\
**Post date:** [August 11, 2020, 9:33am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/3 "2020-08-11T09:33:58Z")

</div>

You sure 😛

The logs are being collected using filebeat but without any modules. As elastic is running within the k8s cluster everything passed to stdout and stderr is being automatically collected by filebeat.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 11, 2020, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/4 "2020-08-11T09:36:03Z")

</div>

You might want to look at the Elasticsearch module, it'll stop the multiline logs from being split like that.

Once that is done you can look at what's causing the error.

---

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\
**Post date:** [August 11, 2020, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/5 "2020-08-11T10:10:51Z")

</div>

Not sure whats wrong.  
But should this be working?

```auto
    setup.ilm.enabled: false
    filebeat.autodiscover.providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths: ["/var/log/containers/*-${data.kubernetes.container.id}.log"]
        multiline.pattern: '^[[:space:]]'
        multiline.negate: false
        multiline.match: after
        exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines
    filebeat.modules:
    - module: elasticsearch
    processors:
    - add_host_metadata:
        netinfo.enabled: false
    - add_cloud_metadata:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/log/containers/"
    - drop_event: #namespaces to be excluded from logging
        when.or:
        {{- range .Values.excludedNamespaces }}
        - equals.kubernetes.namespace: {{ . | quote }}
        {{- end }}

```

---

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\
**Post date:** [August 11, 2020, 10:25am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/6 "2020-08-11T10:25:57Z")

</div>

I think this here could be the issue:

```auto
2020-08-11T10:24:05.656Z ERROR [autodiscover] autodiscover/autodiscover.go:210 Auto discover config check failed for config '{
  "audit": {
    "enabled": true,
    "input": {
      "exclude_lines": [
        "^\\s+[\\-`('.|_]"
      ],
      "multiline": {
        "match": "after",
        "negate": false,
        "pattern": "^[[:space:]]"
      },
      "paths": [
        "/var/log/containers/*-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log"
      ],
      "stream": "all",
      "type": "container"
    }
  },
  "deprecation": {
    "enabled": true,
    "input": {
      "exclude_lines": [
        "^\\s+[\\-`('.|_]"
      ],
      "multiline": {
        "match": "after",
        "negate": false,
        "pattern": "^[[:space:]]"
      },
      "paths": [
        "/var/log/containers/*-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log"
      ],
      "stream": "all",
      "type": "container"
    }
  },
  "gc": {
    "enabled": true,
    "input": {
      "exclude_lines": [
        "^\\s+[\\-`('.|_]"
      ],
      "multiline": {
        "match": "after",
        "negate": false,
        "pattern": "^[[:space:]]"
      },
      "paths": [
        "/var/log/containers/*-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log"
      ],
      "stream": "all",
      "type": "container"
    }
  },
  "module": "elasticsearch",
  "server": {
    "enabled": true,
    "input": {
      "exclude_lines": [
        "^\\s+[\\-`('.|_]"
      ],
      "multiline": {
        "match": "after",
        "negate": false,
        "pattern": "^[[:space:]]"
      },
      "paths": [
        "/var/log/containers/*-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log"
      ],
      "stream": "all",
      "type": "container"
    }
  },
  "slowlog": {
    "enabled": true,
    "input": {
      "exclude_lines": [
        "^\\s+[\\-`('.|_]"
      ],
      "multiline": {
        "match": "after",
        "negate": false,
        "pattern": "^[[:space:]]"
      },
      "paths": [
        "/var/log/containers/*-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log"
      ],
      "stream": "all",
      "type": "container"
    }
  }
}', won't start runner: Can only start an input when all related states are finished: {Id:163938459-66313 Finished:false Fileinfo:0xc000247790 Source:/var/log/containers/elastic-es-data-100-1_elastic-system_elasticsearch-67ff091788282884a43d13156ecbe377050b97cea7ce94a8700def85b451f467.log Offset:1524076 Timestamp:2020-08-11 10:24:05.649762716 +0000 UTC m=+1.048324812 TTL:-1ns Type:container Meta:map[] FileStateOS:163938459-66313}

```

---

<div class="post-metadata">

**Author:** ![raulgs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raulgs/32/68308_2.png) [@raulgs](https://discuss.elastic.co/u/raulgs)\
**Post date:** [August 12, 2020, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/7 "2020-08-12T10:10:43Z")

</div>

@warkolm  
These are the logs errors that flood the log:

```auto
{"type": "server", "timestamp": "2020-08-12T10:08:02,967Z", "level": "WARN", "component": "o.e.x.m.e.l.LocalExporter", "cluster.name": "elastic", "node.name": "elastic-es-master-2", "message": "unexpected error while indexing monitoring document", "cluster.uuid": "OfB8GyE3S-GoLHQr9se2BA", "node.id": "XKKpgvjxR3GMhL4FfK_RCQ" ,
"stacktrace": ["org.elasticsearch.xpack.monitoring.exporter.ExportException: java.lang.IllegalArgumentException: Limit of total fields [1000] in index [.monitoring-es-7-2020.08.12] has been exceeded",
"at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:125) ~[x-pack-monitoring-7.8.1.jar:7.8.1]",
"at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]",
"at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]",
"at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:?]",
"at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]",
"at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]",
"at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) ~[?:?]",
"at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) ~[?:?]",
....
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.8.1.jar:7.8.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]",
"at java.lang.Thread.run(Thread.java:832) ~[?:?]"] }
{"type": "server", "timestamp": "2020-08-12T10:08:02,975Z", "level": "WARN", "component": "o.e.x.m.MonitoringService", "cluster.name": "elastic", "node.name": "elastic-es-master-2", "message": "monitoring execution failed", "cluster.uuid": "OfB8GyE3S-GoLHQr9se2BA", "node.id": "XKKpgvjxR3GMhL4FfK_RCQ" ,
1.49.Final]",

```

Seems like there is an issue with the `xpack.monitoring` package

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2020, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-nodes-create-a-lot-of-logs/244514/8 "2020-09-09T10:10:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.