# Huge logs - how Tuning filebeat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 19, 2023, 7:04pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333 "2023-04-19T19:04:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![pepite](https://avatars.discourse-cdn.com/v4/letter/p/b487fb/32.png) [@pepite](https://discuss.elastic.co/u/pepite)\ **Post date:** [April 19, 2023, 7:04pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/1 "2023-04-19T19:04:43Z")
Hi everybody, I'm french and i m a very newbie with elasticsearch. Elasticsearch version imposed by security team : 7.10.2 I create a cluster like this with dedicate nodes: ``` 2 master node 1 master only eligible node 1 coordinating only node 6 data node 2 data WARM node 2 data COLD node 6 ingest nodes ``` I have to ingest 130 Go of many logs in one time per day of one application (application is load balanced on 32 servers) in production. n reality its the log of the day before. I retrieve an archive of log the day after. I ingest the log xxxxxx.log-2023-04-16 in ELK the day 17th April 2024 For example : ``` the logs are created the 16th April An archive is done in the night of the 16th April. I have to ingest the log of the 16th April the 17th April, the day after also ``` The first day was April, the 14th and the indexing was very very slowly. Also I need to tune πŸ˜‰ correctly Cluster side : ``` Shard : 1 primary, 1 replica Refresh_interval : 30 s ``` Filebeat side : ```auto 9 input log type output elasticsearch hosts : --> i have put all the ingest nodes loadbalance : true --> is it the good practice ? bulk_max_size : 8192 --> can i increase bulk_max_size ? worker : 8 --> can i increase ? until what ? queue.mem.events : --> how calculate the good number ? ``` Could you help me to correctly configure my filebeat please ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 20, 2023, 11:44pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/2 "2023-04-20T23:44:12Z")
I would start by increasing the number of primary shards to 4 or 5 and see if that helps. > [@pepite](#): > > Elasticsearch version imposed by security team : 7.10.2 Please note that version is [EOL](https://www.elastic.co/support/eol) and no longer supported, you should be looking to upgrade as a matter of urgency. It seems odd that the security team would impose an EOL version with known bugs and security issues. ---
**Author:** ![pepite](https://avatars.discourse-cdn.com/v4/letter/p/b487fb/32.png) [@pepite](https://discuss.elastic.co/u/pepite)\ **Post date:** [April 21, 2023, 7:01pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/3 "2023-04-21T19:01:34Z")
> [@warkolm](#): > > would start by increasing the number of primary shards to 4 or 5 and see if that helps Hi thanks for the answer πŸ™‚ I already have an index template attached to an.ilm.policy. I modify the index template by adding the index.number.shards and index.number.replicas then i create an index but it was ko 🀣. I will retry monday but maybe this we it will work this weekend Elk is up :+). Surprise for monday 🀣 I already said for the 7.10.2 EOL but i m in state administration and the decision takes times. An other idea ? πŸ™‚ Thanks. ---
**Author:** ![pepite](https://avatars.discourse-cdn.com/v4/letter/p/b487fb/32.png) [@pepite](https://discuss.elastic.co/u/pepite)\ **Post date:** [April 24, 2023, 8:03pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/4 "2023-04-24T20:03:45Z")
Hi @warkolm Ok my big index was creating with 4 primary shards πŸ™‚ It seems to be better when i watch the indexing time API, if my request was correct. But i m afraid of 2 things with 4 shards : - the disk volume will increase no ? - i would like use shrink api in WARM node to reduce the volume, but first a copy of old index is creating, then i will have 2 BIG indexes. so it requires 2 x the volume of the old index. is it correct ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 25, 2023, 10:31pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/5 "2023-04-25T22:31:43Z")
> [@pepite](#): > > - the disk volume will increase no ? Why's that? You aren't storing additional data are you? > [@pepite](#): > > i would like use shrink api in WARM node to reduce the volume, but first a copy of old index is creating, then i will have 2 BIG indexes. so it requires 2 x the volume of the old index. Yes, that's a requirement for shrink. ---
**Author:** ![pepite](https://avatars.discourse-cdn.com/v4/letter/p/b487fb/32.png) [@pepite](https://discuss.elastic.co/u/pepite)\ **Post date:** [April 28, 2023, 12:58pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/6 "2023-04-28T12:58:35Z")
> [@warkolm](#): > > Why's that? You aren't storing additional data are you? Hi, No only my indices πŸ˜‰ > [@warkolm](#): > > Yes, that's a requirement for shrink. I will test πŸ˜‰ The old indices when shrinking is deleted automatically ? I would like to shrink only one index and reallocate this one on dedicated WARM node with custom attiribute, but for the moment i can't apply index template πŸ˜‰ > [@Index template - exclude index seems not working](https://discuss.elastic.co/t/index-template-exclude-index-seems-not-working/331060): > > Hi everybody. I dont find the correct syntax to exclude one index of an index pattern in index template 2 index template slight_smile 1st { "order": 0, "index\_patterns": ["\*\_\*","-tdir\_business\_prod-\*"], "settings": { "index.lifecycle.name": "Hot\_toWarm\_toCold\_policy", "index.refresh\_interval": "60s", "index.number\_of\_shards": 1, "index.number\_of\_replicas": 1 } } 2nd { "order": 1, "index\_patterns": ["tdir\_business\_prod-\*"], "settings": { "index.lifec… ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 26, 2023, 2:59pm UTC](https://discuss.elastic.co/t/huge-logs-how-tuning-filebeat/330333/7 "2023-05-26T14:59:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.