Hi,
My platform is windows and I fail to process some big (actually not so big) file with file input plugin.
Below is my file input configuration segment:
input { file { path => ["d:/X/zzzzz.csv"] codec => plain { charset => "Big5" } start_position => "beginning" } }
The testing procedure is:
- I start logstash --verbose (without zzzzz.csv in place), I can see:
Registering file input {:path=>["d:/X/zzzzz.csv"], :level=>:info} No sincedb_path set, generating one based on the file path {:sincedb_path=>"C:\\Users\\Magic/.sincedb_535b3b2cc5d2faa77fadaff61b536b79", :path=>[ "d:/X/zzzzz.csv"], :level=>:info}
- I copy a zzzzz.csv (130000 lines, ~8MB) to the place, nothing happened
sincedb file is not created, either. - I close logstash and sincedb file appear (without any content)
- I start logstash again (zzzzz.csv is in place and sincedb file is blank), nothing happen again.
- I close logstash again, and sincedb file content become " 0 0 0" (no file identifier!!)
The situation happen usually (but not always) on my computer especially with big files.
No matter I put data file before logstash start or copy it to the place for logstash to discovery.
Does anyone encounter this situation?
Does file input plugin has any limitation?
(But actually I successfully processed this 8MB file maybe 1 or 2 times....)
Update:
If I reduce the file to 4MB (70000 lines), it can be processed successfully every time.