When does the pending count become 0

Elastic Stack Beats
1

hi I do not speak English well. sorry
my servers log file create 100/1min every day and delete all 12:00 pm
but registry json not show flag "remove" and log.json, number.json just keeps getting bigger.
i wonder When does the 'store' pending count become 0
my server is os : centos7 with filebeat 8.4 and use filesteam
and i complied source filebeat gitbub.

filebeat/input/filestream/internal/input-logfile/store.go

------------org-------------------

func (r *resource) Finished() bool { return r.pending.Load() == 0 }

------------modify--------------

func (r *resource) Finished() bool {
              fmt.Println(r.pending.Load()) 
              return r.pending.Load() == 0 }

and
i removed file or dir
but r.pending.Load is not 0, it is 2
and no matter how much time passes ,it does not change 0

So nothing was deleted when clean.go

i wonder what module is clean or update store instance

2

Please share the configuration you're using.

3

i used when testing

- type: filestream
  paths:
    - /home/test/file/*
  ignore_older: 4m
  close.reader.on_eof: true

or

- type: filestream
  paths:
    - /home/test/file/*
  ignore_older: 4m
  close.reader.on_eof: true
  close.on_state_change.removed: true
  clean_remov: true
4

hi @mtojek

i have question about this
i used config that close.on_state_change.inactive: 2m

  1. when stop hvester due to config, why store memmory ttl is 1800000000000
    (when stop havester, All actions are normal(transport logstash)

  2. when will it change ttl is '0' after stop havester

  3. If ttl is non-zero store's Pendding increases and ttl is changed to 0 when calling delete method due to clean removed ( pendding: 2, ttl=0)
    then what schedule or condition is the store memory deleted?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.