Infrastructure details :
3 * i3.xlarge instances for ES
1 * t2.medium instance for logstash
Volume type : io1
Software details :
ElasticSearch Docker image : 6.3.0
Logstash : 5.3
Beats : 5.3
Requests : 350/sec
Purpose : logging
We are migrating from manages ElasticService provided by AWS to ES on kubernetes. To send data to ES provisioned on kubernetes we are using logstash. Below is the code snippet :
output {
amazon_es {
hosts => ["managed-aws-es"]
region => "ap-southeast-1"
index => "logstash-%{type}-%{+YYYY.MM.dd}"
flush_size => 10
}
elasticsearch {
hosts => ["kubernetes-es:9200"]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
}
When we start sending the data on both the cluster, it becomes slow.
According to me, Logstash don't manage to send data at both the clusters and that increases backpressure on logstash and logs are shipped slowly to both the ElasticSearch cluster. In my opinion
Backpressure on logstash increases because ElasticSearch is not able to process the incoming requests.
In the logs, I got the following errors :
[2018-08-19T18:03:35,413][WARN ][o.e.d.s.f.s.h.UnifiedHighlighter] The length [1027315] of [http.response.body.keyword] field of [VdBbU2UBRaLJ6sM3Mg7X] doc of [packetbeat-2018.08.19] index has exceeded the allowed maximum of [1000000] set for the next major Elastic version. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!
[2018-08-19T18:03:35,418][WARN ][o.e.d.s.f.s.h.UnifiedHighlighter] The length [1027485] of [response] field of [VdBbU2UBRaLJ6sM3Mg7X] doc of [packetbeat-2018.08.19] index has exceeded the allowed maximum of [1000000] set for the next major Elastic version. This maximum can be set by changing the [index.highlight.max_analyzed_offset] index level setting. For large texts, indexing with offsets or term vectors is recommended!
[2018-08-19T18:03:40,353][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][805] overhead, spent [411ms] collecting in the last [1.1s]
[2018-08-19T18:03:43,377][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][808] overhead, spent [447ms] collecting in the last [1s]
[2018-08-19T18:03:45,378][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][810] overhead, spent [505ms] collecting in the last [1s]
[2018-08-19T18:03:50,603][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][815] overhead, spent [428ms] collecting in the last [1.2s]
[2018-08-19T18:03:57,720][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][822] overhead, spent [430ms] collecting in the last [1.1s]
[2018-08-19T18:04:11,777][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][836] overhead, spent [560ms] collecting in the last [1s]
[2018-08-19T18:04:16,056][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][840] overhead, spent [389ms] collecting in the last [1.2s]
[2018-08-19T18:04:23,114][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][847] overhead, spent [364ms] collecting in the last [1s]
[2018-08-19T18:04:24,250][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-2] [gc][848] overhead, spent [408ms] collecting in the last [1.1s]
I've already gone through the threads regarding the issue but could not find the solution
This is a puzzle to me, I've spent so much time to get this fixed but always out of luck.
Please suggest.
