# Filebeat poor performance when publishing existing file to ES **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 6, 2017, 10:41am UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044 "2017-02-06T10:41:28Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\ **Post date:** [February 6, 2017, 10:41am UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/1 "2017-02-06T10:41:28Z")
Hello, I am using filebeat to load existing log file directly into ES. Sample config: ```auto spool_size : 4049 publish_async: true filebeat.prospectors: - paths: ["/data/test.log"] harvester_buffer_size: 163840 fields_under_root: true fields: type: "test1_log" output.elasticsearch: worker: 8 bulk_max_size : 4096 hosts: ["localhost:9200"] pipelines: - pipeline: test-pipeline when.equals: type: "test1_log" ``` ```auto pipeline: { "description" : "Test1 Log", "processors" : [ { "grok" : { "field": "message", "patterns": [ "%{TIMESTAMP_ISO8601:date} %{NUMBER:reqtime} %{NUMBER:http code} %{IP:ip} %{GREEDYDATA:text}" ] }, "date": { "field": "date", "target_field": "date", "formats": ["yyyy-MM-dd HH:mm:ss"] }, "date_index_name" : { "field" : "date", "index_name_prefix" : "test1-", "date_rounding" : "d" } } ] } ``` I tried both 1-server ES setup (default) and 2-servers setup. Each server has 2x 16core CPUs. When I start filebeat, it appends log lines into ES with speed about 4000/second. filebeat process consumes about 15% of CPU core, java (elasticsearch) consumes about 120-150% of CPU. Disks are not overloaded. So it looks like system resources are mostly idle, but loading speed is low. Anything I can tune to improve publishing speed until I hit some resource constrain on my server (CPU, disks)? Thanks in advance. ---
**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\ **Post date:** [February 6, 2017, 10:59am UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/2 "2017-02-06T10:59:53Z")
This thread looks similar: > ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 6, 2017, 3:48pm UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/3 "2017-02-06T15:48:12Z")
please properly format logs and config files using the `` button. Which filebeat version are you using? In 5.x it must say: `filebeat.spool_size:` and `filebeat.publish_async`. With the configuration you have right now load-balancing is not active. I'd recommend having the `filebeat.spooler_size` a multitude of workers time `bulk_max_size`, so batches are split into sub-batches to be properly processed concurrently. This works even is publish\_async is disabled. e.g.: ```auto filebeat.prospectors: - paths: ["/data/test.log"] fields_under_root: true pipeline: "test-pipeline" # spooler size = 2 * 8 * 4096 => split batch into 16 sub-batches to be send concurrently filebeat.spool_size: 65536 # experimental feature, let's first test without it # filebeat.publish_async: true output.elasticsearch: worker: 8 bulk_max_size: 4096 hosts: ["localhost:9200"] # if fields.pipeline is not available in event, no pipeline will be used pipeline: "%{[pipeline]}" ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 6, 2017, 3:49pm UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/4 "2017-02-06T15:49:49Z")
Be carefull to not overload elasticsearch internal pipelines (to big batches or to many workers). In this case some events might not be indexed yet and must be retried by filebeat. This potentially slows down indexing. Check your filebeat log files. ---
**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\ **Post date:** [February 6, 2017, 6:48pm UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/5 "2017-02-06T18:48:35Z")
I was using 5.1.2 and 5.2.0. Yes, I replaced "spool\_size" with "filebeat.spool\_size" and speed improved up to 15000 lines/second! Thanks a lot! PS: shouldn't filebeat report an error when I use (nonexistent) "spool\_size" rather than "filebeat.spool\_size" to avoid such a confusion? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 7, 2017, 1:17am UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/6 "2017-02-07T01:17:12Z")
As throughput for beats very much depends on indexing performance in ES, you might wanna play with 'worker', 'spool\_size' and 'bulk\_max\_size' a little and see if you can adapt throughput somewhat more. Do you have `publish_async` enabled? > PS: shouldn't filebeat report an error when I use (nonexistent) "spool\_size" rather than "filebeat.spool\_size" to avoid such a confusion? We're constantly improving configuration loading. See [go-ucfg](https://github.com/elastic/go-ucfg). Unfortunately we can not detect typos or putting configs on the wrong level yet. Related tickets [#10](https://github.com/elastic/go-ucfg/issues/10), [#11](https://github.com/elastic/go-ucfg/issues/7), [#6](https://github.com/elastic/go-ucfg/issues/6). ---
**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\ **Post date:** [February 7, 2017, 7:51am UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/7 "2017-02-07T07:51:19Z")
> [@steffens](#): > > Do you have publish\_async enabled? No, I removed it per your advice above. If I add it back, performance improves slightly. > [@steffens](#): > > We're constantly improving configuration loading. Unfortunately we can not detect typos or putting configs on the wrong level yet. I see, this is very sad thought understandable. Thanks for you help! ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [February 7, 2017, 1:18pm UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/8 "2017-02-07T13:18:30Z")
using publish\_async filebeat prepares some batches in memory for sending, that is there is some slight latency-overhead if setting is disabled. I'd keep it disabled if possible. You can also try to increase the spool\_size and workers and see if performance still improves somewhat, but maybe you're close to hitting a limit. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2017, 1:18pm UTC](https://discuss.elastic.co/t/filebeat-poor-performance-when-publishing-existing-file-to-es/74044/9 "2017-03-07T13:18:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.