Filebeat 6.1 performance

(Andrew) #1

I transfer postgres logs with Filebeat PostgreSQL module directly to Elasticsearch 6.1. But bandwidth is limited. How I can tuning my config for max performance?
My log:
2018-02-13T12:51:33Z INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=967973984 beat.memstats.memory_alloc=812186752 beat.memstats.memory_total=5431355861888 filebeat.harvester.open_files=1 filebeat.harvester.running=1 libbeat.config.module.running=1 libbeat.output.write.bytes=234873621 libbeat.pipeline.clients=2 libbeat.pipeline.queue.acked=231424 registrar.states.current=1 registrar.states.update=231424 registrar.writes=99

If I understand, this is how many events transfered to ES per 30 sec.

This is my config /etc/filebeat/filebeat.yml
path: ${path.config}/modules.d/.yml
reload.enabled: false "filebeat-%{[beat.version]}"
setup.template.pattern: "filebeat-
host: ""
hosts: ["", "", ""]
index: "filebeat-pg_postgresql-9.5-main.log-%{+yyyy.MM.dd}"
worker: 6
bulk_max_size: 4096
loadbalance: true

Module config /etc/filebeat/modules.d/postgresql.yml
- module: postgresql
enabled: true
var.paths: ["/var/log/postgresql/postgresql-9.5-main.log"]

(Andrew Kroh) #2

If you are trying to optimize for a bandwidth limited link then perhaps you should test the impact of data compression by setting the output.elasticsearch.compression_level option to enable gzip.


(Andrew) #3

I have 1Gbps link, filebeat use only 70Mbps.
I check with compression_level: 4, but any changes in log

2018-02-16T07:43:38Z INFO Non-zero metrics in the last 30s: beat.memstats.gc_next=100961584 beat.memstats.memory_alloc=89754624 beat.memstats.memory_total=19207924664 filebeat.harvester.open_files=1 filebeat.harvester.running=1 libbeat.config.module.running=1 libbeat.output.write.bytes=21602486 libbeat.pipeline.clients=2 libbeat.pipeline.queue.acked=239616 registrar.states.current=1 registrar.states.update=241664 registrar.writes=113

(Andrew) #4

Maybe I need tuning my ES config?
/etc/elasticsearch/elasticsearch.yml mycluster ${HOSTNAME} /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.bind_host: 0
http.port: 9200
transport.tcp.port: 9300

node.master: false true
node.ingest: true

discovery.zen.minimum_master_nodes: 2
transport.profiles.default.port: 9300-9400

indices.requests.cache.size: 10%

# HOT node
node.attr.box_type: hot

(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.