Filebeat is not closing files and open_files count keeps on increasing

[abhishekkene]

The disk space on server shows full and when I checked the Filebeat logs, it was showing the open_files as quite big number, it is continously increasing. The logs are rolling continously and new log file is being created nearly every 2 mins. I 've given the Close_inactive to 1m and it is closing the log file as well but it is not able to close all the log files and few log files are remaining open, which result into increase in disk space. If I restart the Filebeat service then the open_files count goes down and disk space on server as well. Can you please help me in identifing the issue. Please find the current configuration of my filebeat.yml and ymls under input.d folder.

/etc/filebeat/filebeat.yml:::

filebeat.config:
inputs:
enabled: true
path: inputs.d/*yml
reload.enabled: true
reload.period: 10s
setup.template.settings:
index.number_of_shards: 1

/etc/filebeat/inputs.d/modulename.yml::::

• type: log
  paths:
  • /modulename.log
    scan_frequency: 10s
    fields:
    service: modulename
    env: dev
    multiline.type: pattern
    multiline.pattern: ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}
    multiline.negate: true
    multiline.match: after
    close_inactive: 1m
    close_removed: true
    clean_removed: true

[Mario_Castro]

Please, can you paste your configuration in yaml format using common markdown syntax? It's common to have indentation mistakes and nobody will see them if it's not correctly formatted.

Apart from that, if everything is ok, I suggest to start removing config lines one by one to see if it gets fixed this way. If so, you have a mistake in your config (maybe a multiline that never closes)

[abhishekkene]

Please find the yaml files.

/etc/filebeat/filebeat.yml:::

filebeat.config:
  inputs:
    enabled: true
    path: inputs.d/*yml
    reload.enabled: true
    reload.period: 10s
setup.template.settings:
  index.number_of_shards: 1

/etc/filebeat/inputs.d/modulename.yml::::

- type: log
  paths:
    - /modulename.log
  scan_frequency: 10s
  fields:
    service: modulename
    env: dev
  multiline.type: pattern
  multiline.pattern: ^\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d{3}
  multiline.negate: true
  multiline.match: after
  close_inactive: 1m
  close_removed: true
  clean_removed: true

Can you please explain more on your comments about multiline never close. How can I check that?

[Mario_Castro]

I mean try the most bare bones Filebeat config and if the problem persists, Filebeat might have a bug that hasn't been reported yet. It's unlikely because it's the core of Filebeat and other reports will be known yet.

If not, it's something that you are misinterpreting in the config. For example, the multiline pattern might match the very first line of your logs but unexpectedly nothing later. This may be a wrong regex or your logs don't have the shape you are thinking.

Edit: To clarify, I'm not saying that the multiline you have wrote is incorrect, I'm saying that you should try removing config options and check if everything works. Then keep adding config lines one by one until it breaks again.

[abhishekkene]

I 've tried removing the multiline as per your suggestion but no luck.
The open_files count is keep on increasing. I'm not sure whether close_removed or clean_removed is being restricted by some other configuration or any additional configuration is required to work them properly. As in my case, the files are getting removed from server but the filebeat is not able to close them. Please suggest.

[Mario_Castro]

Please, try without any extra configuration. Activate the module, set your paths, execute and check if you keep having the problem

[abhishekkene]

I tried with minimum configuration like having just path and scan_frequency, the count for open_files are increasing quite fast. So it is clear that it is not failing due to other configuration and close_inactive is required and set to 1m in case the log files are getting created very quickly like in my case. But if there is way to make sure close_removed and clean_removed is working properly that would be great. As I can see in my case the even when the files are removed those are not being removed from filebeat registry and hence might be shown as in open_files which actually result in consuming diskspace on server. Please suggest the solution to this issue.

[Mario_Castro]

Ok, now we are talking The reason I was asking you to try a barebones config is because it's often common to misunderstand some config parameter and mess up the system.

Can you provide some more details? I think we need the following:

• Rough number of files being read (and average size of them)
• Rough number of events per file
• Average size of events
• Example log lines of those files.

It can be something very small though, I have seen situations were everything was working like charm until, for some unknown reason, a huge event with a 2 MB single keyword field was generated (maybe from some injection attack) which was producing a huge lag in the entire system.

[abhishekkene]

Please find the details requested:

Number of files being read: 7 (in path: /loglocation/modulename.log)
Number of events : ~15000 events each file
Size of events: Random, mostly 2-3 lines and sometimes stacktrace
Frequency of file creation: 1 new file every 1 or 2 mins
Size of file: 10MB (fixed), once it is reached this size new file will be created

Example:

2021-02-18 23:25:25.360 INFO 24490 --- [hystrix-RestUtility-10] c.g.t.e.m.service.util.RestUtility : Response Body for inventory call from source:::: Ptn Reversal Inventorycall {serviceMaterialId=#######, isSuccess=true, status=SUCCESS, message=Ptn reversal for write off success}
2021-02-18 23:25:25.361 DEBUG 24490 --- [http-nio-9030-exec-3] org.hibernate.SQL : select shoppingca0_.shopping_cart_detail_id as shopping_cart_deta1_62_, shoppingca0_.b2b_circ7_code as b2_62_, shoppingca0_.created_by as created_by3_62_, shoppingca0_.creation_date as creation_date4_62_, shoppingca0_.cust_item_price as cust_item_price5_62_, shoppingca0_.customer_part_number as customer_part_numb6_62_, shoppingca0_.defect_id as defect_id7_62_, shoppingca0_.deleted_quantity as deleted_quantity8_62_, shoppingca0_.dft_defect_pick_flag as dft_defect_pick_fl9_62_, shoppingca0_.dft_shortage_comments as dft_shortage_comm10_62_, shoppingca0_.dmr_item_flag as dmr_item_flag11_62_, shoppingca0_.dmr_quantity_due as dmr_quantity_due12_62_, shoppingca0_.floor_pick as floor_pick13_62_, shoppingca0_.fmi_id as fmi_id14_62_, shoppingca0_.item_id_picked as item_id_picked15_62_, shoppingca0_.item_number_picked as item_number_picke16_62_, shoppingca0_.last_updated_by as last_updated_by17_62_, shoppingca0_.last_updated_date as last_updated_date18_62_, shoppingca0_.last_usage_date as last_usage_date19_62_, shoppingca0_.locator_id as locator_id20_62_, shoppingca0_.material_request_date as material_request_21_62_, shoppingca0_.material_request_location as material_request_22_62_, shoppingca0_.matl_requested_by as matl_requested_by23_62_, shoppingca0_.parts_catalog_item_id as parts_catalog_ite24_62_, shoppingca0_.parts_catalog_item_number as parts_catalog_ite25_62_, shoppingca0_.picked_quantity as picked_quantity26_62_, shoppingca0_.po_line_number as po_line_number27_62_, shoppingca0_.po_number as po_number28_62_, shoppingca0_.repeator_flag as repeator_flag29_62_, shoppingca0_.request_source as request_source30_62_, shoppingca0_.requested_during_outage as requested_during_31_62_, shoppingca0_.requested_quantity as requested_quantit32_62_, shoppingca0_.returned_quantity as returned_quantity33_62_, shoppingca0_.service_item_id as service_item_id34_62_, shoppingca0_.shopping_cart_header_id as shopping_cart_hea45_62_, shoppingca0_.shopping_cart_status as shopping_cart_sta35_62_, shoppingca0_.shopping_cart_status_id as shopping_cart_sta36_62_, shoppingca0_.shortage_comments as shortage_comments37_62_, shoppingca0_.shortage_flag as shortage_flag38_62_, shoppingca0_.shortage_item_id as shortage_item_id39_62_, shoppingca0_.source_inv_code as source_inv_code40_62_, shoppingca0_.stock_locator as stock_locator41_62_, shoppingca0_.subinventory as subinventory42_62_, shoppingca0_.ux_item_flag as ux_item_flag43_62_, shoppingca0_.ux_quantity_due as ux_quantity_due44_62_ from ############################## shoppingca0_ where shoppingca0_.shopping_cart_detail_id=?