Filebeat data.json missing after minor upgrade

Elastic Stack Beats
1

Hi there.

After upgrading from filebeat v7.7.0 to v7.9.2 on a windows 2012 server I no longer have a data.json file in my registry folder. This has been replaced with a file named 'X.json' where X is an almost continuously changing number, counting upwards in increments of ~20'000-30'0000. E.g. the file will be called '1000000.json' and then change to '1024564.json' anywhere from 1-5 seconds later. Then 10539092.json and so on in a seemingly random fashion.

I presume some failure occurred when installing the new version of filebeat. I have tried uninstalling filebeat using the PS script provided, deleting the lingering files in "Program Files" and in "Program Data", and installing as normal from a fresh download. The issue remains.

All files specified by the inputs in filebeat.yml have been reharvested as it appears to have started it's registry from scratch/overwritten/deleted the prior file (prior to my reinstallation attempt). There is no mention of any change of behaviour to the registry files in the documentation for the release or general 7.9 documentation.

In my 'ProgramData -> filebeat -> registry -> filebeat' folder, I currently have the following files:

  • the ever increasingly larger number named X.json
  • active.dat
  • log.json
  • meta.json

Can I please get some advice as to whether this is expected behaviour and what the resolution might be? As it stands, if I have to delete items from the registry as I would have done prior (by deleting a portion of the data.json) I'm unsure as to where to delete these.

I will add requested additional information to this post as required.

Many thanks in advance.

C

2

Reverted back to Filebeat 7.7.0 and works fine, as expected. Went back to 7.9.2 and issue replicates as above.

3

Bump. I also have the same situation and would like to know how to manage this new set of registry files.

Specifically, I need to clean the registry of the progress on one specific log file and then restart filebeat so I can re-read in the entire log file, but with the enormous size and constant changing of the X.json file it's become almost impossible.

4

Also running into this issue with Filebeat > 7.7.0. We use the registry to determine if some automatically-generated files have been ingested yet or not.
Would love to get this fixed.

5

I met a similiar problem using filbeat v7.9.2 on windows. last month, after running for 4-5 days, filebeat stops reading in new files and XXXXXXXX.json appears in the registry directory, with the filename constantly changing. I wonder why would such filename with random numbers appear. What's the machnisim?

6

Bump. Any update from Elastic?

7

Identical issue on 7.10. Need to get information on what files are finished harvested and can be deleted.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.