Hello,
I use filebeat to send slow log to logstash. I use the mysql module for that. But when I try to send an old file, filebeat send only the last slow query of the file. I already tried to delete the registry and I search for a missing configuration in my filebeat.yml, but I found nothing.
My filebeat configuration :
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: "filebeat-test"
# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["filebeat-test"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#================================ Outputs =====================================
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["logstashIP:5044"]
An extract of my filebeat log :
2018-10-11T02:10:32.375-0400 INFO log/input.go:111 Configured paths: [/root/*.log]
2018-10-11T02:10:32.375-0400 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 0
2018-10-11T02:10:32.375-0400 INFO cfgfile/reload.go:122 Config reloader started
2018-10-11T02:10:32.376-0400 INFO log/input.go:111 Configured paths: [/var/log/mysql/error.log* /var/log/mysqld.log*]
2018-10-11T02:10:32.377-0400 INFO log/input.go:111 Configured paths: [/root/*.log]
2018-10-11T02:10:32.377-0400 INFO cfgfile/reload.go:253 Starting 1 runners ...
2018-10-11T02:10:32.377-0400 INFO input/input.go:87 Starting input of type: log; ID: 18267570244159057816
2018-10-11T02:10:32.377-0400 INFO input/input.go:87 Starting input of type: log; ID: 15544384872414468624
2018-10-11T02:10:32.377-0400 INFO cfgfile/reload.go:214 Loading of config files completed.
2018-10-11T02:10:32.378-0400 INFO log/harvester.go:228 Harvester started for file: /root/test.log
2018-10-11T02:11:02.376-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":13}},"total":{"ticks":20,"time":{"ms":24},"value":0},"user":{"ticks":10,"time":{"ms":11}}},"info":{"ephemeral_id":"a20f239b-72f3-4c4e-8071-30fdc78aea92","uptime":{"ms":30008}},"memstats":{"gc_next":5662672,"memory_alloc":3122544,"memory_total":6951832,"rss":15351808}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1},"output":{"events":{"acked":3,"batches":2,"total":3},"read":{"bytes":12},"type":"logstash","write":{"bytes":1026}},"pipeline":{"clients":4,"events":{"active":0,"filtered":5,"published":3,"retry":2,"total":8},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":1,"update":8},"writes":7},"system":{"cpu":{"cores":1},"load":{"1":0.06,"15":0.05,"5":0.03,"norm":{"1":0.06,"15":0.05,"5":0.03}}}}}}
2018-10-11T02:11:32.376-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":1}},"total":{"ticks":20,"time":{"ms":2},"value":20},"user":{"ticks":10,"time":{"ms":1}}},"info":{"ephemeral_id":"a20f239b-72f3-4c4e-8071-30fdc78aea92","uptime":{"ms":60008}},"memstats":{"gc_next":5662672,"memory_alloc":3290088,"memory_total":7119376}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"pipeline":{"clients":4,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.04,"15":0.05,"5":0.03,"norm":{"1":0.04,"15":0.05,"5":0.03}}}}}}
Thanks in advance !
Olivier