Filebeate can not remove the state of log in registry file

Elastic Stack Beats
1

The Filebeat Version I used is 6.8.6. Running in Linux.
The filebeat.yml I used is as fllow:

#========================= Filebeat global options ============================
filebeat.config:
  inputs:
    enabled: true
    path: /etc/filebeat/filebeat-1/*.yml
    reload.enabled: true
    reload.period: 1s
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["192.168.113.128:9200"]

I set the filebeat to load external configuration files.
For example, here is the contents about the external files.

/etc/filebeat/filebeat-1/
└── filebeat-config-1.yml

filebeat-config-1.yml

- type: log
  paths:
    - /tmp/test-1.txt
  close_inactive: 2m
  scan_frequency: 3s
  clean_inactive: 11m
  ignore_older: 5m

Then I start the filebeat with this command in the filebeat directory.

./filebeat -e -c filebeat.yml

After the filebeat started, I write some message to the monitored file.

echo "Hello world" > /tmp/test-1.txt

In filebeat log, I can see the message

2020-07-23T10:10:33.594+0800	INFO	log/harvester.go:255	Harvester started for file: /tmp/test-1.txt

It also create a new registry file, writing the state of this file.

{
  "source": "/tmp/test-1.txt",
  "offset": 12,
  "timestamp": "2020-07-23T10:10:34.838957384+08:00",
  "ttl": 660000000000,
  "type": "log",
  "meta": null,
  "FileStateOS": {
    "inode": 72384636,
    "device": 64768
  }
}

2 minutes later, the harvester is closed, and filebeat updates the timestamp in the registry file.

2020-07-23T10:12:38.609+0800	INFO	log/harvester.go:280	File is inactive: /tmp/test-1.txt. Closing because close_inactive of 2m0s reached.

{
  "source": "/tmp/test-1.txt",
  "offset": 12,
  "timestamp": "2020-07-23T10:12:38.609247928+08:00",
  "ttl": 660000000000,
  "type": "log",
  "meta": null,
  "FileStateOS": {
    "inode": 72384636,
    "device": 64768
  }
}

However, after watting nearly 20 minutes, I found that filebeat didn't clean the state of this file in the registry file. I didn't change anything to this file.

[root@localhost linux_shell]# stat /tmp/test-1.txt 
  File: ‘/tmp/test-1.txt’
  Size: 12        	Blocks: 8          IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 72384636    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: unconfined_u:object_r:user_tmp_t:s0
Access: 2020-07-23 10:10:33.592645059 +0800
Modify: 2020-07-23 10:10:33.442645069 +0800
Change: 2020-07-23 10:10:33.442645069 +0800
 Birth: -

2020-07-23T10:38:14.552+0800	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":2900,"time":{"ms":48}},"total":{"ticks":3640,"time":{"ms":78},"value":3640},"user":{"ticks":740,"time":{"ms":30}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":6},"info":{"ephemeral_id":"b9025b62-7e97-4b7d-a3b2-eabe85923453","uptime":{"ms":1800026}},"memstats":{"gc_next":4369568,"memory_alloc":2844384,"memory_total":83034448}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":30},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0,"15":0.05,"5":0.01,"norm":{"1":0,"15":0.0125,"5":0.0025}}}}}}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.