I can't send older log

Elastic Stack Beats
1

Hello,

I use filebeat to send slow log to logstash. I use the mysql module for that. But when I try to send an old file, filebeat send only the last slow query of the file. I already tried to delete the registry and I search for a missing configuration in my filebeat.yml, but I found nothing.

My filebeat configuration :

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
name: "filebeat-test"

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["filebeat-test"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["logstashIP:5044"]

An extract of my filebeat log :

2018-10-11T02:10:32.375-0400	INFO	log/input.go:111	Configured paths: [/root/*.log]
2018-10-11T02:10:32.375-0400	INFO	crawler/crawler.go:82	Loading and starting Inputs completed. Enabled inputs: 0
2018-10-11T02:10:32.375-0400	INFO	cfgfile/reload.go:122	Config reloader started
2018-10-11T02:10:32.376-0400	INFO	log/input.go:111	Configured paths: [/var/log/mysql/error.log* /var/log/mysqld.log*]
2018-10-11T02:10:32.377-0400	INFO	log/input.go:111	Configured paths: [/root/*.log]
2018-10-11T02:10:32.377-0400	INFO	cfgfile/reload.go:253	Starting 1 runners ...
2018-10-11T02:10:32.377-0400	INFO	input/input.go:87	Starting input of type: log; ID: 18267570244159057816 
2018-10-11T02:10:32.377-0400	INFO	input/input.go:87	Starting input of type: log; ID: 15544384872414468624 
2018-10-11T02:10:32.377-0400	INFO	cfgfile/reload.go:214	Loading of config files completed.
2018-10-11T02:10:32.378-0400	INFO	log/harvester.go:228	Harvester started for file: /root/test.log
2018-10-11T02:11:02.376-0400	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":13}},"total":{"ticks":20,"time":{"ms":24},"value":0},"user":{"ticks":10,"time":{"ms":11}}},"info":{"ephemeral_id":"a20f239b-72f3-4c4e-8071-30fdc78aea92","uptime":{"ms":30008}},"memstats":{"gc_next":5662672,"memory_alloc":3122544,"memory_total":6951832,"rss":15351808}},"filebeat":{"events":{"added":8,"done":8},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":1,"starts":1},"reloads":1},"output":{"events":{"acked":3,"batches":2,"total":3},"read":{"bytes":12},"type":"logstash","write":{"bytes":1026}},"pipeline":{"clients":4,"events":{"active":0,"filtered":5,"published":3,"retry":2,"total":8},"queue":{"acked":3}}},"registrar":{"states":{"cleanup":1,"current":1,"update":8},"writes":7},"system":{"cpu":{"cores":1},"load":{"1":0.06,"15":0.05,"5":0.03,"norm":{"1":0.06,"15":0.05,"5":0.03}}}}}}
2018-10-11T02:11:32.376-0400	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":{"ms":1}},"total":{"ticks":20,"time":{"ms":2},"value":20},"user":{"ticks":10,"time":{"ms":1}}},"info":{"ephemeral_id":"a20f239b-72f3-4c4e-8071-30fdc78aea92","uptime":{"ms":60008}},"memstats":{"gc_next":5662672,"memory_alloc":3290088,"memory_total":7119376}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"pipeline":{"clients":4,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.04,"15":0.05,"5":0.03,"norm":{"1":0.04,"15":0.05,"5":0.03}}}}}}

Thanks in advance !
Olivier

2

What's the contents of your module configurations?

The filebeat log output suggest look for these files only: /root/*.log, /var/log/mysql/error.log*, /var/log/mysqld.log*. Is the slowlog fileset enabled?

3

Hi,

In my mysql module, I only configure the path. To try an old log, I configured it to "/root/*.log". My slowlog fileset is enable. I receive Only the first ans last query of the file.

If my slow log comes from another server, can this be a problem?

Sorry for my english if I make some mistakes(it is not my native language).

4

If my slow log comes from another server, can this be a problem?

Depends where/how the data is written and stored. Is the file local to filebeat or on another host? Is it shared via Network Shares (which often can be a problem)?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.