Files crawling count issue in filebeat [8.6.2]

Elastic Stack Beats
1

We were previously using the filebeat version 7.17.3 and we keep on monitoring the filebeat daily to prepare the reports. Recently, we have upgraded the filebeat to version 8.6.2 and we have observed two things:

  • We observed that the filebeat log files are rotated as ndjson files where as in the 7.17.3, the file wasn't in the ndjson format.
**7.17.3 - File_rotation**
![filebeat_files-7.17.3|690x362](upload://mCMxezsb7wMBHjk2bqEVrfDLxK1.png)

**8.6.2 - File_rotation**
![filebeat_files-8.6.2|690x164](upload://tFVx9P3vHcMSgoAeWomI84W6X02.png)
  • We observed that after the upgrade, we couldn't see the open and running files that the filebeat is crawling
**7.17.3 - Count_of_files_crawled**
![files_crawling-7.17.3|690x161](upload://bwSBm12pYM1HFoDHkizgYabfgzz.png)

**8.6.2 - Count_of_files_crawled**
![files_crawling-8.6.2|690x89](upload://dEOQiX1QakaBMebd3TBLOoDNgZR.png)

Attached are the images of the observations. Please help me understanding the above.

I have tried using the command "tail -f filebeat-2023-10-05-1" to retrieve the number of files crawling in the filebeat as we are used to fetch the data in 7.17.3 version but we got an error saying the file isn't existing. When we have checked for the available files, we see that the files are in ndjson format. So, I have tried the command "tail -f filebeat-20231005.ndjson" and see that the open and running files as 0 instead of the number of files that the filebeat is crawling.

Thanks & regards,
Rohit.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.