Filebeat 5.0alpha3/4 doesn't clean registry and crashes


(Gabry993) #1

Hi,
I'm facing a couple of problem when running filebeat (both alpha 3 and 4) on a windows server 2003.
I'm trying to send the content of a log file which is written continuously and it's rotated often.
This is how the rotation works: after reaching a certain size, the file "logxxx.log" get renamed in "logxxx-timestamp.log" and a new "logxxx.log" is created.
As the file is rotated very often (and I am only on testing, the real environment creates a lot of log file with the same logic) I would like to controll the registry size, so this is my prospector config:

- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - yyy\logxxx.log
  exclude_lines: ["^\\s*$"]
  include_lines: ["^([a-z]|[A-Z])"]
  ignore_older: 1m
  document_type: log_xxx
  registry:
    clean_inactive: 2m

I'm not sure if this is a working config, but I tried everything! The registry won't be cleaned, even if in the log file of filebeat I found many entries like these:

DBG State removed for yyy\logxxx.log because of older: %!s(MISSING)

Moreover, after running for a couple of hour, filebeat crashes and I get this error which I can't understand, but I guess is related with the registry clean up:

panic: runtime error: slice bounds out of range

goroutine 16 [running]:
panic(0x8c3a80, 0x1125a010)
        /usr/local/go/src/runtime/panic.go:481 +0x326
github.com/elastic/beats/filebeat/input/file.(*States).Cleanup(0x113b6f60, 0xf84
75800, 0xd)
        /go/src/github.com/elastic/beats/filebeat/input/file/state.go:93 +0x388
github.com/elastic/beats/filebeat/prospector.(*ProspectorLog).Run(0x113b37c0)
        /go/src/github.com/elastic/beats/filebeat/prospector/prospector_log.go:5
3 +0x84
github.com/elastic/beats/filebeat/prospector.(*Prospector).Run(0x113b8420)
        /go/src/github.com/elastic/beats/filebeat/prospector/prospector.go:129 +
0x290
github.com/elastic/beats/filebeat/crawler.(*Crawler).Start.func1(0x113de0f0, 0x0
, 0x113b8420)
        /go/src/github.com/elastic/beats/filebeat/crawler/crawler.go:71 +0x102
created by github.com/elastic/beats/filebeat/crawler.(*Crawler).Start
        /go/src/github.com/elastic/beats/filebeat/crawler/crawler.go:72 +0x41d

And these are the last filebeat log lines:

2016-08-01T18:27:41+02:00 INFO Registry file updated. 94 states written.
2016-08-01T18:27:41+02:00 INFO Run prospector
2016-08-01T18:27:41+02:00 DBG Start next scan
2016-08-01T18:27:41+02:00 DBG Check file for harvesting: yyy\logxxx.log
2016-08-01T18:27:41+02:00 DBG Update existing file for harvesting: yyy\logxxx.log, offset: 1350352
2016-08-01T18:27:41+02:00 DBG No updates needed, file yyy\logxxx.log is already harvested.
2016-08-01T18:27:41+02:00 DBG State removed for yyy\logxxx.log because of older: %!s(MISSING)
2016-08-01T18:27:41+02:00 DBG State removed for yyy\logxxx.log because of older: %!s(MISSING)
2016-08-01T18:27:41+02:00 DBG State removed for yyy\logxxx.log because of older: %!s(MISSING)

Thank you for your attention!
Gabriele


(ruflin) #2

Two things here:

Can you please check the most recent builds to see if the problems persist?


(Gabry993) #3

I'm currently using the alpha4. I will try the nightly as soon as I can!
Thank you!


(Gabry993) #4

Everything went perfect with the nightly!
Thank you for all your work and support!


(system) #5

This topic was automatically closed after 21 days. New replies are no longer allowed.