# Fluentd to elastic
**URL:** https://discuss.elastic.co/t/fluentd-to-elastic/190427
**Category:** Elasticsearch
**Created:** [July 15, 2019, 5:12am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427 "2019-07-15T05:12:32Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 5:12am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/1 "2019-07-15T05:12:32Z")
I am shipping logs using fluentd in k8s cluster i see a bunch of the following messages and logs stop flowing to ES
warn]: [elasticsearch] failed to write data into buffer by buffer overflow action=:block
Any thoughts or solution
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 5:13am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/2 "2019-07-15T05:13:50Z")
Sounds like there might be a problem with the Elasticsearch cluster. Is there anything in the Elasticsearch logs?
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 5:24am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/3 "2019-07-15T05:24:06Z")
I see a bunch of the following
[2019-07-15T05:20:32,226][INFO][o.e.m.j.JvmGcMonitorService] [elasticsearch-logging-0] [gc][39] overhead, spent [403ms] collecting in the last [1s]
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 5:25am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/4 "2019-07-15T05:25:36Z")
just another data point meteric beat continues to work in the same setup
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 5:26am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/5 "2019-07-15T05:26:27Z")
What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)? Which version of Elasticsearch are you using?
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 5:32am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/6 "2019-07-15T05:32:42Z")
ES 6.2.5
{
"\_nodes" : {
"total" : 2,
"successful" : 2,
"failed" : 0
},
"cluster\_name" : "kubernetes-logging",
"timestamp" : 1563168679582,
"status" : "yellow",
"indices" : {
"count" : 97,
"shards" : {
"total" : 481,
"primaries" : 481,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 1,
"max" : 5,
"avg" : 4.958762886597938
},
"primaries" : {
"min" : 1,
"max" : 5,
"avg" : 4.958762886597938
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 58508426,
"deleted" : 6
},
"store" : {
"size\_in\_bytes" : 17602171642
},
"fielddata" : {
"memory\_size\_in\_bytes" : 0,
"evictions" : 0
},
"query\_cache" : {
"memory\_size\_in\_bytes" : 0,
"total\_count" : 0,
"hit\_count" : 0,
"miss\_count" : 0,
"cache\_size" : 0,
"cache\_count" : 0,
"evictions" : 0
},
"completion" : {
"size\_in\_bytes" : 0
},
"segments" : {
"count" : 3060,
"memory\_in\_bytes" : 82584599,
"terms\_memory\_in\_bytes" : 49103002,
"stored\_fields\_memory\_in\_bytes" : 13437920,
"term\_vectors\_memory\_in\_bytes" : 0,
"norms\_memory\_in\_bytes" : 3027392,
"points\_memory\_in\_bytes" : 1778397,
"doc\_values\_memory\_in\_bytes" : 15237888,
"index\_writer\_memory\_in\_bytes" : 0,
"version\_map\_memory\_in\_bytes" : 0,
"fixed\_bit\_set\_memory\_in\_bytes" : 0,
"max\_unsafe\_auto\_id\_timestamp" : 1563168028913,
"file\_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 2,
"data" : 2,
"coordinating\_only" : 0,
"master" : 2,
"ingest" : 2
},
"versions" : [
"6.2.4"
],
"os" : {
"available\_processors" : 16,
"allocated\_processors" : 16,
"names" : [
{
"name" : "Linux",
"count" : 2
}
],
"mem" : {
"total\_in\_bytes" : 32434601984,
"free\_in\_bytes" : 8051560448,
"used\_in\_bytes" : 24383041536,
"free\_percent" : 25,
"used\_percent" : 75
}
},
"process" : {
"cpu" : {
"percent" : 1
},
"open\_file\_descriptors" : {
"min" : 852,
"max" : 1236,
"avg" : 1044
}
},
"jvm" : {
"max\_uptime\_in\_millis" : 715637,
"versions" : [
{
"version" : "1.8.0\_161",
"vm\_name" : "OpenJDK 64-Bit Server VM",
"vm\_version" : "25.161-b14",
"vm\_vendor" : "Oracle Corporation",
"count" : 2
}
],
"mem" : {
"heap\_used\_in\_bytes" : 1055361600,
"heap\_max\_in\_bytes" : 2075918336
},
"threads" : 179
},
"fs" : {
"total\_in\_bytes" : 44120965120,
"free\_in\_bytes" : 6204526592,
"available\_in\_bytes" : 3916115968
},
"plugins" : [
{
"name" : "ingest-user-agent",
"version" : "6.2.4",
"description" : "Ingest processor that extracts information from a user agent",
"classname" : "org.elasticsearch.ingest.useragent.IngestUserAgentPlugin",
"extended\_plugins" : ,
"has\_native\_controller" : false,
"requires\_keystore" : false
},
{
"name" : "ingest-geoip",
"version" : "6.2.4",
"description" : "Ingest processor that uses looksup geo data based on ip adresses using the Maxmind geo database",
"classname" : "org.elasticsearch.ingest.geoip.IngestGeoIpPlugin",
"extended\_plugins" : ,
"has\_native\_controller" : false,
"requires\_keystore" : false
}
],
"network\_types" : {
"transport\_types" : {
"netty4" : 2
},
"http\_types" : {
"netty4" : 2
}
}
}
}
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 5:34am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/7 "2019-07-15T05:34:31Z")
You seem to have a good amount of RAM but a default 1GB heap. I would recommend you increase the heap size of the nodes to e.g. 4GB to see if that helps.
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 5:48am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/8 "2019-07-15T05:48:55Z")
> [@Soumitra\_Ghosh](#):
>
> 3243460
Hi i see file to change in a native rpm install but if i am getting it from a docker image
[k8s.gcr.io/elasticsearch:v6.2.5](http://k8s.gcr.io/elasticsearch:v6.2.5)
where would one make such a change.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 5:51am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/9 "2019-07-15T05:51:15Z")
I have not used that image so am not sure. Would however recommend using [the official docker images](https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html).
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 6:09am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/10 "2019-07-15T06:09:46Z")
just to confirm it will be the following to a say 2048m to start with for example
- "ES\_JAVA\_OPTS=-Xms512m -Xmx512m"
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 6:11am UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/11 "2019-07-15T06:11:47Z")
That is 512 MB heap, which is smaller than what you have now.
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 6:17pm UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/12 "2019-07-15T18:17:13Z")
> [@Christian\_Dahlqvist](#):
>
> heap
Hi i see that my heap used is 1GB less than max so how does this help? Anything else to try?
thanks
"mem" : {
"heap\_used\_in\_bytes" : 1055361600,
"heap\_max\_in\_bytes" : 2075918336
},
"threads" : 179
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 15, 2019, 6:20pm UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/13 "2019-07-15T18:20:44Z")
That is across 2 nodes, so each node only have 1GB heap configured. Update your jvm.options files and increase the heap size to 2GB or 4GB.
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 15, 2019, 7:34pm UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/14 "2019-07-15T19:34:18Z")
added the another node and updated the mem - output below but still same error
"mem" : {
"heap\_used" : "1.9gb",
"heap\_used\_in\_bytes" : 2073279480,
"heap\_max" : "5.8gb",
"heap\_max\_in\_bytes" : 6233260032
},
"threads" : 307
---
### Author:  [@Soumitra\_Ghosh](https://discuss.elastic.co/u/Soumitra_Ghosh)
#### Post date: [July 16, 2019, 1:37pm UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/15 "2019-07-16T13:37:17Z")
Anything else as the last idea did not work.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 13, 2019, 1:37pm UTC](https://discuss.elastic.co/t/fluentd-to-elastic/190427/16 "2019-08-13T13:37:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.