# Need lots of Help with ELK

**URL:** <https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528>\
**Category:** Elasticsearch\
**Created:** [August 21, 2016, 9:29am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528 "2016-08-21T09:29:33Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 9:29am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/1 "2016-08-21T09:29:33Z")

</div>

Hi all,  
I am new to ELK,  
I have a system that collect system logs from appx 150 servers,  
all send syslog to one machine:  
its users 16 cores and 32gb of memory - networking running on a bond with 2 nics.  
its running on centos 7 all updated to the latest version.  
the problem is that its very very slow to respond specifically kibana.  
it takes very very long initilize the kibana index, and after it does, it take very long to show the data  
or search in it.  
after reading many forums  
I'm totally clue less, what am I doing wrong?  
at first it was running ok, but after a week or so it became painfully slow  
I'll post my configuration in the next post because i can't do more than 5k lines.

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 9:31am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/2 "2016-08-21T09:31:40Z")

</div>

> input {  
> udp {  
> port =\> 514  
> type =\> syslog  
> codec =\> json  
> }  
> udp {  
> port =\> 3515  
> codec =\> "json"  
> type =\> "WindowsLog"  
> }  
> }

> filter {  
> if [type] == "syslog" {  
> grok {  
> match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" }  
> add\_field =\> ["received\_at", "%{@timestamp}"]  
> add\_field =\> ["received\_from", "%{host}"]  
> match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{COMBINEDAPACHELOG}" }  
> }  
> }  
> if [type] == "WindowsLog" {  
> json {  
> source =\> "message"  
> }  
> if [SourceModuleName] == "EventLog" {

> ```
> mutate {
> replace => ["message", "%{Message}"]
> }
> }
> mutate {
> remove_field => ["Message"]
> }
> 
> ```
> 
> }  
> date {  
> match =\> ["MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]  
> }

> mutate {  
> remove\_field =\> ["geoip.ip","geoip.latitude"]  
> remove\_tag =\> ["geoip.ip","geoip.latitude"]  
> }  
> }

> output {  
> elasticsearch {  
> hosts =\> "x.x.x.x"  
> index =\> "logstash-%{+YYYY.MM.dd}"  
> }

> }

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 9:32am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/3 "2016-08-21T09:32:15Z")

</div>

this is my elasticsearch.yml

> # ======================== Elasticsearch Configuration =========================
> 
> # 
> 
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> 
> # Before you set out to tweak and tune the configuration, make sure you
> 
> # understand what are you trying to accomplish and the consequences.
> 
> # 
> 
> # The primary way of configuring a node is via this file. This template lists
> 
> # the most important settings you may want to configure for a production cluster.
> 
> # 
> 
> # Please see the documentation for further information on configuration options:
> 
> # [http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html)
> 
> # 
> 
> # ---------------------------------- Cluster -----------------------------------
> 
> # 
> 
> # Use a descriptive name for your cluster:
> 
> # 
> 
> # cluster.name: my-application
> 
> # 
> 
> # ------------------------------------ Node ------------------------------------

> # Use a descriptive name for the node:
> 
> # 
> 
> # node.name: node-1
> 
> # 
> 
> # Add custom attributes to the node:
> 
> # 
> 
> # node.rack: r1
> 
> # 
> 
> # ----------------------------------- Paths ------------------------------------
> 
> # 
> 
> # Path to directory where to store the data (separate multiple locations by comma):
> 
> # 
> 
> # path.data: /path/to/data
> 
> # 
> 
> # Path to log files:
> 
> # 
> 
> # path.logs: /path/to/logs
> 
> # 
> 
> # ----------------------------------- Memory -----------------------------------
> 
> # 
> 
> # Lock the memory on startup:
> 
> # 
> 
> # bootstrap.mlockall: true
> 
> # 
> 
> # Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory
> 
> # available on the system and that the owner of the process is allowed to use this limit.
> 
> # 
> 
> # Elasticsearch performs poorly when the system is swapping the memory.
> 
> # 
> 
> # ---------------------------------- Network -----------------------------------
> 
> # 
> 
> # Set the bind address to a specific IP (IPv4 or IPv6):
> 
> # 
> 
> network.host: x.x.x.x
> 
> # 
> 
> # Set a custom port for HTTP:
> 
> # 
> 
> http.port: 9200
> 
> # 
> 
> # For more information, see the documentation at:
> 
> # [http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html)
> 
> # 
> 
> # --------------------------------- Discovery ----------------------------------
> 
> # 
> 
> # Pass an initial list of hosts to perform discovery when new node is started:
> 
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> 
> # 
> 
> # discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> 
> # 
> 
> # Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
> 
> # 
> 
> # discovery.zen.minimum\_master\_nodes: 3
> 
> # 
> 
> # For more information, see the documentation at:
> 
> # [http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html)
> 
> # 
> 
> # ---------------------------------- Gateway -----------------------------------
> 
> # 
> 
> # Block initial recovery after a full cluster restart until N nodes are started:
> 
> # 
> 
> # gateway.recover\_after\_nodes: 3
> 
> # 
> 
> # For more information, see the documentation at:
> 
> # [http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html)
> 
> # 
> 
> # ---------------------------------- Various -----------------------------------
> 
> # 
> 
> # Disable starting multiple nodes on a single system:
> 
> # 
> 
> # node.max\_local\_storage\_nodes: 1
> 
> # 
> 
> # Require explicit names when deleting indices:
> 
> # 
> 
> # action.destructive\_requires\_name: true
> 
> index.number\_of\_shards: 20  
> index.number\_of\_replicas: 0  
> bootstrap.mlockall: true

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 9:32am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/4 "2016-08-21T09:32:46Z")

</div>

also, note that top showes that the server is not on high loads  
I've disabled swapping as mentioned in lots of forums,  
i don't know what to do anymore, please please assist, if anymore information is needed please tell me and i'll provide.  
p.s. - i've changed the address with x.x.x.x for obvious resons.  
many, many thanks in Advance!!!

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 21, 2016, 11:28am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/5 "2016-08-21T11:28:44Z")

</div>

How much data are you indexing per day? How long do you intend to keep data in the cluster? Which version of Elasticsearch and Logstash are you using? What type of storage does the node have?

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 12:11pm UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/6 "2016-08-21T12:11:54Z")

</div>

I want to achieve 10gigs a day now Its writing about 3gigs a day , I'm using centralized storage (Dell Equalogic) im on the latest version of all (es-2.3.3,ls-2.3.2,kibana-4.5.4) and i want to hold data for 90 days (write now i have 20 days already). also, I'm not on a cluster, I run it on one powerful machine (16 cores, 32gb Mem) and while "top"ing i don't see that the machine on a large load and system utilization, also i don't reach even 10% of the Storage IOPs capability also, no CPU wait either. so as you see, I am kind of clueless...

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 21, 2016, 1:53pm UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/7 "2016-08-21T13:53:56Z")

</div>

> [@Nimrod\_levy](#):
>
> index.number\_of\_shards: 20

If those are the volumes, why have you set the default shard count to 20??? Having lots of small shards carries overhead and can be very inefficient, for indexing as well as querying. Given the volumes you have mentioned I would recommend setting this to 1 in order to achieve a shard size between a few GB and a few tens of GB in size.

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 21, 2016, 2:03pm UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/8 "2016-08-21T14:03:17Z")

</div>

I've changed it to 1  
and sitll doesn't do anything 😕  
moreover, now almost every search falls down to the timeout

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 21, 2016, 5:28pm UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/9 "2016-08-21T17:28:36Z")

</div>

This change will only affect future indices, so will not help immediately. How much data do you have in the cluster? How many shards?

---

<div class="post-metadata">

**Author:** ![Nimrod\_levy](https://avatars.discourse-cdn.com/v4/letter/n/8491ac/32.png) [@Nimrod\_levy](https://discuss.elastic.co/u/Nimrod_levy)\
**Post date:** [August 22, 2016, 6:48am UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/10 "2016-08-22T06:48:24Z")

</div>

Christian, Thanks for your help, I had this Eurika Moment and I have found the problem, here is what I've done.  
(I am working on a CentOS7 so it applies to Centos7/RHEL7 in Debians that may differ.

1. edit /etc/security/limits.conf  
I have Added:  
elasticsearch - nofile 65535  
elasticsearch - memlock unlimited

2. edit /etc/sysconfig/elasticsearch  
I have Added:  
MAX\_LOCKED\_MEMORY=unlimited  
MAX\_OPEN\_FILES=65536  
ES\_HEAP\_SIZE=16G

3. edit /etc/elasticsearch/elasticsearch.yml  
and added:  
index.number\_of\_shards: 1  
bootstrap.mlockall: true

4. I have commented the swap partition from /etc/fstab  
and issued "swapoff -a"

and suddenly kibana and elasticsearch started to fly...  
basically i've let the machine use all of its resources - i have only one node - but a powerful one.  
well that occurred to me since you said that i have lots of small syslog messages so, writing it to memory fast and  
than queuing it to the disk is the solution, Christian, Many Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:26pm UTC](https://discuss.elastic.co/t/need-lots-of-help-with-elk/58528/11 "2017-07-05T22:26:17Z")

</div>