# Filebeat consuming high I/O operations for smaller files **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 24, 2022, 11:18am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105 "2022-06-24T11:18:22Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [June 24, 2022, 11:18am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/1 "2022-06-24T11:18:22Z")
I have the following architecture: 1. Files (\*.json) of ~ **1-5 KB** gets written to a folder throughout the day where file beat is scanning with a frequency of **10s**. 2. A json file once written in the file system will never be updated. Below is the configuration: ```auto filebeat.inputs: - type: log multiline.pattern: '^\{' multiline.negate: true multiline.match: after multiline.max_lines: 1000000 multiline.timeout: 10s tail_files: false enabled: true paths: - /opt/JSON/20220615/*.json fields: type: json fields_under_root: true # This option is to uniquely identify a file by filebeat. # In our use case a file will never be updated after creation and files will be stored in network share system. # So a file path can be used as unique identifier in our case. Below configuration option serves that purpose. file_identity.path: ~ ### Harvester Options harvester_limit: 4096 ignore_older: 1h scan_frequency: 10s ### Harvester closing options close_eof: true ### State options clean_inactive: 2h ``` Following are few questions which I need few inputs: 1. Is it ideal to have too many small files for filebeat to scan or it is advised to append the content into a single json files of ~10MB and rotate ? With the current scheme and configuration, I am observing high I/O operation. 2. Even though the **ignore\_older** is set to **1h** , does it implies that for all files which are already indexed and older than 1h, the file handler will be opened for these to identify whether it has undergone any change and then the file handler is closed or whether the file changed information is available from the inode itself? 3. Any other optimization is recommended? ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [June 27, 2022, 6:56am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/2 "2022-06-27T06:56:38Z")
Can anyone suggest if I am missing something? ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [June 29, 2022, 9:40am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/3 "2022-06-29T09:40:04Z")
Requesting for a feedback on this ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [July 4, 2022, 12:56pm UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/4 "2022-07-04T12:56:12Z")
> [@diptesh2007](#): > > - Is it ideal to have too many small files for filebeat to scan or it is advised to append the content into a single json files of ~10MB and rotate ? With the current scheme and configuration, I am observing high I/O operation. I would say that it is better to collect from less files, as there is less overhead in the system. If you can control this, try to log to less files. But you may end up having similar problems if the file rotates too frequently. > [@diptesh2007](#): > > - Even though the **ignore\_older** is set to **1h** , does it implies that for all files which are already indexed and older than 1h, the file handler will be opened for these to identify whether it has undergone any change and then the file handler is closed or whether the file changed information is available from the inode itself? With `ignore_older: 1h`, any file whose last modification happened more than one hour ago is going to be ignored and no file handler is going to be open for them. It doesn't matter if it has been indexed or not. If the file was being read, you also need to use `close_inactive` to ensure that the file handler is closed. You can read more about this setting in [Log input | Filebeat Reference [8.3] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.3/filebeat-input-log.html#filebeat-input-log-ignore-older) > [@diptesh2007](#): > > Any other optimization is recommended? Have you tried to increase [`registry.flush`](https://www.elastic.co/guide/en/beats/filebeat/8.3/configuration-general-options.html#_registry_flush)? This controls with what frequency filebeat persists its state on disk. Increasing this value may help when you have many files. ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [July 12, 2022, 5:57pm UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/5 "2022-07-12T17:57:43Z")
> Have you tried to increase [`registry.flush`](https://www.elastic.co/guide/en/beats/filebeat/8.3/configuration-general-options.html#_registry_flush)? This controls with what frequency filebeat persists its state on disk. Increasing this value may help when you have many files. This is currently set as ```auto filebeat.registry.flush: 60s ``` > With `ignore_older: 1h` , any file whose last modification happened more than one hour ago is going to be ignored and no file handler is going to be open for them. It doesn't matter if it has been indexed or not. Does the file modification time of a file resides in inode which is why the filebeat is able to filter out the older files without actually opening the file handler? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [July 14, 2022, 9:40am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/6 "2022-07-14T09:40:25Z")
> [@diptesh2007](#): > > Does the file modification time of a file resides in inode which is why the filebeat is able to filter out the older files without actually opening the file handler? Yes, Filebeat relies on the filesystem to obtain the modification time. It can be checked without actually opening the file. ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [July 17, 2022, 11:24am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/7 "2022-07-17T11:24:27Z")
> Yes, Filebeat relies on the filesystem to obtain the modification time. It can be checked without actually opening the file If that is the case, when filebeat is running with large number of input files, I wonder why my EFS burst I/O credit balance is decreasing rapidly. Any pointers? > But you may end up having similar problems if the file rotates too frequently. What is the standard recommendation to rotate the log file? Size based rotation or time based rotation? ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [July 20, 2022, 5:41am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/8 "2022-07-20T05:41:57Z")
Any thoughts on the standard rotation policies? ---
**Author:** ![diptesh2007](https://avatars.discourse-cdn.com/v4/letter/d/e79b87/32.png) [@diptesh2007](https://discuss.elastic.co/u/diptesh2007)\ **Post date:** [August 4, 2022, 7:59am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/9 "2022-08-04T07:59:39Z")
Trying to keep this thread alive in case someone would like to suggest the recommended polcies ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 1, 2022, 10:00am UTC](https://discuss.elastic.co/t/filebeat-consuming-high-i-o-operations-for-smaller-files/308105/10 "2022-09-01T10:00:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.