Filebeat - Non-zero metrics in the last 30s

Elastic Stack Beats
,
1

The file update has not been detected, but the file has been updated and the ES index has not been written.
log

2021-06-04T12:39:30.519Z	INFO	instance/beat.go:304	Setup Beat: filebeat; Version: 7.12.1
2021-06-04T12:39:30.519Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://192.168.2.117:19200
2021-06-04T12:39:30.519Z	INFO	eslegclient/connection.go:99	elasticsearch url: http://xxx:19200
2021-06-04T12:39:30.520Z	INFO	[publisher]	pipeline/module.go:113	Beat name: 9625679f4bf6
2021-06-04T12:39:30.520Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2021-06-04T12:39:30.521Z	INFO	instance/beat.go:468	filebeat start running.
2021-06-04T12:39:30.521Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-06-04T12:39:30.521Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-06-04T12:39:30.521Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-06-04T12:39:30.521Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 2
2021-06-04T12:39:30.520Z	INFO	[publisher]	pipeline/module.go:113	Beat name: 9625679f4bf6
2021-06-04T12:39:30.520Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2021-06-04T12:39:30.521Z	INFO	instance/beat.go:468	filebeat start running.
2021-06-04T12:39:30.521Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-06-04T12:39:30.521Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-06-04T12:39:30.521Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-06-04T12:39:30.521Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 2
2021-06-04T12:39:30.521Z	INFO	log/input.go:157	Configured paths: [/var/yoocar/deploys/mall-order-serve/log/custom/*.log]
2021-06-04T12:39:30.521Z	INFO	log/input.go:157	Configured paths: [/deploys/mall-order-serve/log/custom/*.log]
2021-06-04T12:39:30.521Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6240586857139569280)
2021-06-04T12:39:30.521Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6240586857139569280)
2021-06-04T12:39:30.522Z	INFO	log/input.go:157	Configured paths: [/var/yoocar/deploys/mall-order-serve/log/operation/*.log]
2021-06-04T12:39:30.522Z	INFO	log/input.go:157	Configured paths: [/deploys/mall-order-serve/log/operation/*.log]
2021-06-04T12:39:30.522Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 4301979234238418463)
2021-06-04T12:39:30.522Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 2

2021-06-04T12:40:00.525Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"9625679f4bf6b7b36c6777f04653f2c738664387941ae0bb1680ecfb775b10a7"},"cpuacct":{"id":"9625679f4bf6b7b36c6777f04653f2c738664387941ae0bb1680ecfb775b10a7","total":{"ns":314027765}},"memory":{"id":"9625679f4bf6b7b36c6777f04653f2c738664387941ae0bb1680ecfb775b10a7","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":29114368}}}},"cpu":{"system":{"ticks":10,"time":{"ms":15}},"total":{"ticks":90,"time":{"ms":100},"value":0},"user":{"ticks":80,"time":{"ms":85}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"1a84d77b-0cc5-46cd-9002-03b2b543e216","uptime":{"ms":30059}},"memstats":{"gc_next":18926528,"memory_alloc":9789744,"memory_sys":75056128,"memory_total":45533768,"rss":75001856},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":2},"load":{"1":1.69,"15":1.33,"5":1.59,"norm":{"1":0.845,"15":0.665,"5":0.795}}}}}}

yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /deploys/mall-order-serve/log/custom/*.log
  fileds:
    type: custom
  json.keys_under_root: true
  json.overwrite_keys: true
- type: log
  enabled: true
  paths:
    - /deploys/mall-order-serve/log/operation/*.log
  fileds:
    type: operation
  json.keys_under_root: true
  json.overwrite_keys: true

output.elasticsearch:
  hosts: ["xxx:19200"]
  index: "java_log_%{[fields.type]}_%{+yyyy.MM.dd}"
  indices:
    - index: "java_log_custom_%{+yyyy.MM.dd}"
      when.equals:
          fields.type: custom
    - index: "java_log_operation_%{+yyyy.MM.dd}"
      when.equals:
          fields.type: operation
setup.ilm.enabled: false                     
setup.template.name: java_log         
setup.template.pattern: java_log_*         
setup.template.overwrite: true
setup.template.enabled: true

What should I do ？ Thinks

2

Could you please share the debug logs of Filebeat? Run it with ./filebeat -e -d "*".

BTW, there is a typo in your configuration fileds should be fields. It is probably unrelated to your problem.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.