# Suggestion on Elasticsearch scaling and performance for log management
**URL:** https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548
**Category:** Elasticsearch
**Created:** [September 15, 2019, 5:25pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548 "2019-09-15T17:25:41Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@Blason](https://discuss.elastic.co/u/Blason)
#### Post date: [September 15, 2019, 5:25pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/1 "2019-09-15T17:25:41Z")
Hi Folks,
I know this is a very vague question and might have answered a couple of times before, but wanted another pair of eyes to look at and vet?
I am planning to build log management/Security Analytics solution and will be collecting logs from around 100 devices comprises for [Servers/Routers/Switches/Firewalls] which I believe should not generate more than 15-20 GB per day
I am planning with 4 nodes
- 1xLogstash - 24GB
- 2xES nodes [cluster] - 32 GB each/ having 3 TB of Diskspace
- 1xwazuh/OSSEC node for accepting messages from Server sending it to ES directly with 16Gb RAM
Now my queries are -
- How many shards should be configured? 5 is enough?
- What should be HEAP\_SIZE on ES? Considering 32 GB -\> 16GB is enough?
- Can I install Kibana on Primary elasticsearch node? Or do I need to install Kibana on a different server?
- And Kibana will/should connect to the primary node of the cluster?
- Similarly, Logstash will send data to the primary node as well?
- Considering future growth and shard numbers I can add more ES node in the cluster; right?
- Any other optimization tips are really appreciated
TIA
Blason R
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 15, 2019, 6:12pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/2 "2019-09-15T18:12:59Z")
> [@Blason](#):
>
> - How many shards should be configured? 5 is enough?
1 sounds enough (and this is the default in recent versions). 5 sounds like too many. Use [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) to move to a new index when the current one reaches a reasonable size (say 40GB).
> [@Blason](#):
>
> - What should be HEAP\_SIZE on ES? Considering 32 GB -\> 16GB is enough?
If your machines have 32GB of RAM then 16GB (50%) is the [absolute maximum allowed](https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html). You may get better performance with a smaller heap.
> [@Blason](#):
>
> - Can I install Kibana on Primary elasticsearch node? Or do I need to install Kibana on a different server?
There's no such thing as a "primary node". Both nodes are equal from Elasticsearch's point of view, and you can send data and searches to either.
> [@Blason](#):
>
> - Considering future growth and shard numbers I can add more ES node in the cluster; right?
Yes. Note that you cannot build a fault-tolerant cluster with only two nodes - you need at least three master-eligible nodes for resilience. But after that you can add data nodes as needed. You might also want to segregate your data nodes into a [hot/warm architecture](https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x).
---
### Author:  [@Blason](https://discuss.elastic.co/u/Blason)
#### Post date: [September 15, 2019, 6:22pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/3 "2019-09-15T18:22:35Z")
Super!! Only confusing part for me is shards? Wondering how 1 shard would suffice my need?
My ultimate goal here is to achieve cluster with
- Fault tolerant
- Scalable
- And of course Optimally used
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 15, 2019, 6:34pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/4 "2019-09-15T18:34:17Z")
I don't really understand the question. What is making you think that 1 shard per index will not be enough?
---
### Author:  [@Blason](https://discuss.elastic.co/u/Blason)
#### Post date: [September 15, 2019, 6:43pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/5 "2019-09-15T18:43:49Z")
If I am not wrong one shard probably will not spawn on multiple nodes, correct? and considering the future growth if data volume increases; if I introduce 2 mode nodes 1 shard will not be enough, right?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 15, 2019, 6:47pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/6 "2019-09-15T18:47:40Z")
Ok I think I see. By "number of shards" I mean the number set by the [`index.number_of_shards` setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings). `1` is the default for that and that sounds reasonable for your case. But for fault tolerance each shard must have a replica, and this is controlled by the independent [`index.number_of_replicas` setting](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#dynamic-index-settings). `1` is the default for that too, meaning that each shard will have a copy on both nodes, and that sounds good for you too.
---
### Author:  [@gavenkoa](https://discuss.elastic.co/u/gavenkoa)
#### Post date: [September 16, 2019, 1:27pm UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/7 "2019-09-16T13:27:16Z")
> [@Blason](#):
>
> 1xLogstash - 24GB
Why do you need Logstash? I think it's a bit pure software because of Ruby interpreter...
ES has ingest nodes since 5.x and supports 30 or so input filters.
Unless you need to enhance IP addresses with GEO data (as I know there is no such ES filter) I don't understand why do you need to waste 24 GiB of RAM when it can be ES ingest node.
- [A New Way To Ingest - Part 1 | Elastic Blog](https://www.elastic.co/blog/new-way-to-ingest-part-1)
- [A New Way To Ingest - Part 2 | Elastic Blog](https://www.elastic.co/blog/new-way-to-ingest-part-2)
- [Should I use Logstash or Elasticsearch ingest nodes? | Elastic Blog](https://www.elastic.co/blog/should-i-use-logstash-or-elasticsearch-ingest-nodes)
---
### Author:  [@Blason](https://discuss.elastic.co/u/Blason)
#### Post date: [September 17, 2019, 3:06am UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/8 "2019-09-17T03:06:33Z")
Ingest node however I am not so sure about the ingest node can start listening on any port for incoming data from my network devices? From servers it would work since I will be installing shippers but what about Network devices?
---
### Author:  [@gavenkoa](https://discuss.elastic.co/u/gavenkoa)
#### Post date: [September 17, 2019, 8:44am UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/9 "2019-09-17T08:44:08Z")
> [@Blason](#):
>
> can start listening on any port for incoming data from my network devices
> **[Packetbeat: Network Analytics Using Elasticsearch | Elastic](https://www.elastic.co/beats/packetbeat)**
>
> Monitor application performance by analyzing network protocols like HTTP, DNS, MySQL, Postgres, and more in real time and integrate with Elasticsearch....
## Pick a Protocol. Or Build Your Own
Packetbeat is also a library. It supports many application layer protocols, from database to key-value stores to HTTP and low-level protocols. Choose the one you need or add your own by submitting a pull request.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 15, 2019, 8:44am UTC](https://discuss.elastic.co/t/suggestion-on-elasticsearch-scaling-and-performance-for-log-management/199548/10 "2019-10-15T08:44:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.