I am sending kubernetes pod logs from filebeat to logstash via kafka plugin. Filebeat, Logstash and kafka running as pods in the single host.
filebeat ==> Kafka ==> Logstash
when i checked the network bandwidth, the filebeat is sending around 50 Mb/s, where as logstash also receiving almost same. To avoid bottleneck's, I used a simple logstash config with no filters ( as shown below)
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash
namespace: "{{ .Values.namespace }}"
data:
logstash.conf: |-
input {
kafka {
bootstrap_servers => "{{ .Values.config.kafkaService }}:{{ .Values.config.kafkaServicePort }}"
codec => plain { charset => "UTF-8" }
consumer_threads => 1
topics => ["T1-LOG","T2-LOG","T3-LOG"]
decorate_events => true
}
}
output {
stdout { codec => dots }
}
logstash.yml: |-
http.host: "0.0.0.0"
xpack.monitoring.enabled: false
queue.type: memory
But when i sent the logs directely from filbeat to logstash (filebeat ===> logstash), I am getting around 6Mb/s between logstash and filebeat.
apiVersion: v1
kind: ConfigMap
metadata:
name: logstash
namespace: "{{ .Values.namespace }}"
data:
logstash.conf: |-
input {
beats {
port => 5044
}
}
output {
stdout { codec => dots }
}
logstash.yml: |-
http.host: "0.0.0.0"
xpack.monitoring.enabled: false
queue.type: memory
What could be the possible reason for low througput when sending logs from filbeat to logstash ? how could I improve this performance ?