Bug for sincedb with gz file

Elastic Stack Logstash
1

Hi
I'm facing with issue for *.xml.gz files generally issue was reproduced on gz compress files.
Even on the latest version of logstash 8.2.1
When I recovered logstash service with sincedb configuration it seems that queue (with files are processing once again).
The sincedb mechanism is designed to prevent such situations. In my opinion sincedb file is also badly written. Only last one line has a fully name of file and then after parsing the file it disappears. This is quite an important feature (and it would be nice to have coverage for compressed logs). Please fix it.

6443347212 0 64768 74046 1653479246.869928
6443347214 0 64768 73362 1653479249.458039
6443347216 0 64768 76342 1653479251.817318
6443347218 0 64768 76000 1653479253.557689
6443347220 0 64768 74892 1653479255.799229
6443347222 0 64768 75621 1653479257.4988608
6443106966 0 64768 75564 1653479259.364306
6443347226 0 64768 77324 1653479261.313035
6443347228 0 64768 75636 1653479263.741848
6443347230 0 64768 75246 1653479265.805993
6443347232 0 64768 75009 1653479267.907712
6443347235 0 64768 76160 1653479269.9910839
6443347237 0 64768 77122 1653479272.467931
6443347240 0 64768 73616 1653479274.500069
6443347242 0 64768 73624 1653479276.724158
6443347244 0 64768 76277 1653479279.183906
6443347246 0 64768 73283 1653479281.760788
6443347248 0 64768 73315 1653479283.950932
6443347250 0 64768 75366 1653479286.382374
6443347252 0 64768 73616 1653479289.040152
6443347254 0 64768 76145 1653479291.547809
6443347256 0 64768 76333 1653479294.038769
6443347258 0 64768 76901 1653479296.130042
6443347260 0 64768 76592 1653479298.502644
6443347262 0 64768 74610 1653479300.670604
6443348418 0 64768 75824 1653479302.671524
6443348421 0 64768 77399 1653479305.086715
6443348434 0 64768 79737 1653479307.621136
6443348446 0 64768 76225 1653479309.9653728
6443348448 0 64768 77433 1653479312.1789799
6443348464 0 64768 76415 1653479314.736031
6443348466 0 64768 80364 1653479316.9134922
6443348468 0 64768 78276 1653479319.3346992
6443348470 0 64768 77413 1653479321.901298
6443348474 0 64768 77112 1653479324.234174
6443348476 0 64768 80971 1653479326.536776
6443348705 0 64768 83687 1653479329.428642 /opt/data/input/A20220525.0530+0200-0535+0200_HSS40.xml.gz
2

If you think this is a bug you should open a bug report in the github repository of the file input plugin with the issue you got and how someone can reproduce it.

It is the best way to make sure that this is really an issue and that it will be looked at.

3

Why do you care about the filename? The sincedb tracks files using the inode and device numbers. If a file is renamed from foo.log to foo.log.1 these do not change, so the input will know not to re-read it just because it has been rotated. There are issues with the sincedb, but I do not think this is one of them.

4

Yes then I will do it that

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.