Logstash Configuration: [https://gist.github.com/hagait/23f4b2bc614a4c4acbb6](https://gist.github.com/hagait/23f4b2bc614a4c4acbb6)
Elasticsearch
configuration: [https://gist.github.com/hagait/ba3684048abe2f9219b8](https://gist.github.com/hagait/ba3684048abe2f9219b8)
Thank you for the support!
Regards,
Hagai
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 5, 2015, 11:32am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/2 "2015-02-05T11:32:55Z")
What is the question?...
15k inserts per sec per node is actually quite nice.
Are your index sharded? If you write to one index only, you write to
maximum of x nodes where x is the number of shards of that index. Since
shards of the same index can co-exist on one node, check if you are
spanning writes.
Use local disks - never EBS, and if you really care about writing speeds
use SSDs.
Other than that, Mike did an excellent write up on the subject :
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Thu, Feb 5, 2015 at 1:26 PM, Hagai T [hagai.tob@gmail.com](mailto:hagai.tob@gmail.com) wrote:
> Hi Guys,
>
> We use Elasticsearch as our tracking system of our products in a dynamic
> to track performance.
> Searching of this data is being used by small group of users (10-12) in
> the company to measure performance.
>
> In our current environment, we see limit of 15,000 documents being
> inserted per second without the ability to scale.
>
> Some information on the current setup and the flow:
>
> _- Tracking servers_
> 8 x Apache servers behind Amazon ELB which serve empty html files so it
> tracks the parameters given and writes it to Apache access log.
> on each server, we also have Logstash which configured to read this access
> log file and send data to Elasticsearch cluster.
>
> \*- Elasticsearch Cluster: \*
> 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> process - 30GB Heap size
> 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> processes each with 30GB Heap size.
>
> Additional information on the Cluster:
> [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt)
>
> Cluster health:
> [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#file-gistfile1-txt)
>
> Logstash Configuration:
> [gist:23f4b2bc614a4c4acbb6 · GitHub](https://gist.github.com/hagait/23f4b2bc614a4c4acbb6)
>
> Elasticsearch configuration:
> [gist:ba3684048abe2f9219b8 · GitHub](https://gist.github.com/hagait/ba3684048abe2f9219b8)
>
> Thank you for the support!
> Regards,
> Hagai
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuuedagMgkbe4JoUGWg2DT1pFMkmdRKS3Rp4f4mwq6ntg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuuedagMgkbe4JoUGWg2DT1pFMkmdRKS3Rp4f4mwq6ntg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Hagai\_T](https://discuss.elastic.co/u/Hagai_T)
#### Post date: [February 5, 2015, 11:51am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/3 "2015-02-05T11:51:06Z")
Hi Itamar, thank you for the reply.
This is 15k inserts totally and not for one host in the cluster.
Yes, we have 15 sharding whiting one index. shards are spreaded on the
nodes equally (automatically by Elasticsearch cluster).
We currently use general purpose SSD and not Ephemeral storage.
In addition, I see a lot of thread pool bulk rejections from the
Elasticsearch side.
On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko wrote:
> What is the question?...
>
> 15k inserts per sec per node is actually quite nice.
>
> Are your index sharded? If you write to one index only, you write to
> maximum of x nodes where x is the number of shards of that index. Since
> shards of the same index can co-exist on one node, check if you are
> spanning writes.
>
> Use local disks - never EBS, and if you really care about writing speeds
> use SSDs.
>
> Other than that, Mike did an excellent write up on the subject :
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/performance-considerations-elasticsearch-indexing/)
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Thu, Feb 5, 2015 at 1:26 PM, Hagai T \<[haga...@gmail.com](mailto:haga...@gmail.com) \\>
> wrote:
>
> > Hi Guys,
> >
> > We use Elasticsearch as our tracking system of our products in a dynamic
> > to track performance.
> > Searching of this data is being used by small group of users (10-12) in
> > the company to measure performance.
> >
> > In our current environment, we see limit of 15,000 documents being
> > inserted per second without the ability to scale.
> >
> > Some information on the current setup and the flow:
> >
> > _- Tracking servers_
> > 8 x Apache servers behind Amazon ELB which serve empty html files so it
> > tracks the parameters given and writes it to Apache access log.
> > on each server, we also have Logstash which configured to read this
> > access log file and send data to Elasticsearch cluster.
> >
> > \*- Elasticsearch Cluster: \*
> > 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> > process - 30GB Heap size
> > 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> > processes each with 30GB Heap size.
> >
> > Additional information on the Cluster:
> > [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt)
> >
> > Cluster health:
> > [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#file-gistfile1-txt)
> >
> > Logstash Configuration:
> > [gist:23f4b2bc614a4c4acbb6 · GitHub](https://gist.github.com/hagait/23f4b2bc614a4c4acbb6)
> >
> > Elasticsearch configuration:
> > [gist:ba3684048abe2f9219b8 · GitHub](https://gist.github.com/hagait/ba3684048abe2f9219b8)
> >
> > Thank you for the support!
> > Regards,
> > Hagai
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 5, 2015, 11:56am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/4 "2015-02-05T11:56:27Z")
I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use of
the snapshot/restore API will provide you HA and DR guarantees.
The rejections you are seeing are due to slow I/O operations, because the
disk is not local. There is a way to have a bigger queue but I'd advise
against that and instead go with a local fast disk.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Thu, Feb 5, 2015 at 1:51 PM, Hagai T [hagai.tob@gmail.com](mailto:hagai.tob@gmail.com) wrote:
> Hi Itamar, thank you for the reply.
>
> This is 15k inserts totally and not for one host in the cluster.
> Yes, we have 15 sharding whiting one index. shards are spreaded on the
> nodes equally (automatically by Elasticsearch cluster).
> We currently use general purpose SSD and not Ephemeral storage.
>
> In addition, I see a lot of thread pool bulk rejections from the
> Elasticsearch side.
>
> On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
> wrote:
>
> > What is the question?...
> >
> > 15k inserts per sec per node is actually quite nice.
> >
> > Are your index sharded? If you write to one index only, you write to
> > maximum of x nodes where x is the number of shards of that index. Since
> > shards of the same index can co-exist on one node, check if you are
> > spanning writes.
> >
> > Use local disks - never EBS, and if you really care about writing speeds
> > use SSDs.
> >
> > Other than that, Mike did an excellent write up on the subject :
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/performance-)
> > considerations-elasticsearch-indexing/
> >
> > --
> >
> > Itamar Syn-Hershko
> > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > Freelance Developer & Consultant
> > [Lucene.NET](http://Lucene.NET) committer and PMC member
> >
> > On Thu, Feb 5, 2015 at 1:26 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> >
> > > Hi Guys,
> > >
> > > We use Elasticsearch as our tracking system of our products in a dynamic
> > > to track performance.
> > > Searching of this data is being used by small group of users (10-12) in
> > > the company to measure performance.
> > >
> > > In our current environment, we see limit of 15,000 documents being
> > > inserted per second without the ability to scale.
> > >
> > > Some information on the current setup and the flow:
> > >
> > > _- Tracking servers_
> > > 8 x Apache servers behind Amazon ELB which serve empty html files so it
> > > tracks the parameters given and writes it to Apache access log.
> > > on each server, we also have Logstash which configured to read this
> > > access log file and send data to Elasticsearch cluster.
> > >
> > > \*- Elasticsearch Cluster: \*
> > > 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> > > process - 30GB Heap size
> > > 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> > > processes each with 30GB Heap size.
> > >
> > > Additional information on the Cluster:
> > > [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt)
> > >
> > > Cluster health: [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#)
> > > file-gistfile1-txt
> > >
> > > Logstash Configuration: [https://gist.github.com/](https://gist.github.com/)
> > > hagait/23f4b2bc614a4c4acbb6
> > >
> > > Elasticsearch configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait/)
> > > ba3684048abe2f9219b8
> > >
> > > Thank you for the support!
> > > Regards,
> > > Hagai
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%
> > > [40googlegroups.com](http://40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zv0toyUEhc%2BynpkH%3DUxf9GVLuvnKdPSGUQcnG1grwpbCw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zv0toyUEhc%2BynpkH%3DUxf9GVLuvnKdPSGUQcnG1grwpbCw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Hagai\_T](https://discuss.elastic.co/u/Hagai_T)
#### Post date: [February 9, 2015, 10:15am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/5 "2015-02-09T10:15:48Z")
Hi,
We were able to identify the bottleneck which seems to be the Logstash
service.
It seems that the Elasticsearch cluster is able to handle 40,000 per second
documents with a 3 ES servers cluster using Java client that was written by
us using SDK with bulk inserts.
The client (written for load testing) is generating JSON format and send it
to Elasticsearch for further processing.
We run the same test with Logstash which reads the JSON format from Apache
access log on a general purpose SSD and managed to achieve maximum of 4,000
requests per second.
With 2 Logstash servers we achieved 8,000 req per second.
Getting rid of the filtering section in logstash configuration file helped
us get to this number. with filtering we achieved only 1,5000-2,000 req per
sec.
I also tried to move the log file to Ephemeral storage but didn't get any
improvement.
We don't have any resources problem in the Logstash server (I/O / CPU) so
it seems like a limit in the file input module or either logstash itself.
I was able to test Logstash performance by creating huge log file 2GB and
starting Logstash to send it's content.
I also did tried with smaller files (4-5MB each) but performance didn't get
any better.
Does it sound reasonable for you guys that I got to a limit of 4,000 req
per second with one Logstash?
If you have any suggestions of how to proceed from here I will be more than
happy to hear that.
If we can't get more from one Logstash, we'll have to develop our own Java
service to do that instead.
_Apache Access log file output example - (already in JSON Format):_
{ "timestamp":"2015-02-09T10:07:48+0000",
"bq\_timestamp":"2015-02-09T10:07:48", "client\_ip":"52.2.11.111",
"client\_port":"80", "latency\_ms":"57", "latency\_sec":"0",
"elb\_status\_code":"200",
"request":"/il.html?e=fpAdOpportunity&w=wfl\_dose&vid=1&vname=compName\_PM&ecpm=8&adid=1814157&media\_file\_type=MEDIA\_FILE\_TYPE&media\_file\_url=MEDIA\_FILE\_URL¤t\_url=%0Ahttp%3A%2F%[2Fu-sd.gga.tv](http://2Fu-sd.gga.tv)%2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau\_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%[252F3ffese.com](http://252F3ffese.com)%26eov%3Deov%0A%09¤t\_main\_vast\_url=MAIN\_VAST\_URL&error\_code=ERROR\_CODE&error\_message=ERROR\_MESSAGE&[q9=dsdase.com](http://q9=dsdase.com)&[apid=dose.com](http://apid=dose.com)&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",
"request\_path":"/il.html", "referer":"-", "user\_agent":"Mozilla/5.0
(redhat-x86\_64-linux-gnu) Siege/3.0.8" }
_Logstash configuration file (for the testing I ran it with root without
any limitations):_
input {
file {
path =\>
"/var/log/httpd/aaa.ddddd.com.logstash-acc.log.[0-9]\*"
codec =\> json
type =\> "tracking"
discover\_interval =\> 1
sincedb\_path =\> "/opt/logstash/httpd-sincedb"
sincedb\_write\_interval =\> 1
}
}
output {
elasticsearch {
workers =\> 1
host =\> "[aaa.dddd.com](http://aaa.dddd.com)"
index =\> "%{request\_path}-logstash-%{+YYYY-MM-dd}"
flush\_size =\> 1000
cluster =\> "video"
codec =\> json
}
}
Your help is appreciated!
Thanks!
On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko wrote:
> I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use of
> the snapshot/restore API will provide you HA and DR guarantees.
>
> The rejections you are seeing are due to slow I/O operations, because the
> disk is not local. There is a way to have a bigger queue but I'd advise
> against that and instead go with a local fast disk.
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Thu, Feb 5, 2015 at 1:51 PM, Hagai T \<[haga...@gmail.com](mailto:haga...@gmail.com) \\>
> wrote:
>
> > Hi Itamar, thank you for the reply.
> >
> > This is 15k inserts totally and not for one host in the cluster.
> > Yes, we have 15 sharding whiting one index. shards are spreaded on the
> > nodes equally (automatically by Elasticsearch cluster).
> > We currently use general purpose SSD and not Ephemeral storage.
> >
> > In addition, I see a lot of thread pool bulk rejections from the
> > Elasticsearch side.
> >
> > On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
> > wrote:
> >
> > > What is the question?...
> > >
> > > 15k inserts per sec per node is actually quite nice.
> > >
> > > Are your index sharded? If you write to one index only, you write to
> > > maximum of x nodes where x is the number of shards of that index. Since
> > > shards of the same index can co-exist on one node, check if you are
> > > spanning writes.
> > >
> > > Use local disks - never EBS, and if you really care about writing speeds
> > > use SSDs.
> > >
> > > Other than that, Mike did an excellent write up on the subject :
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/performance-)
> > > considerations-elasticsearch-indexing/
> > >
> > > --
> > >
> > > Itamar Syn-Hershko
> > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > > Freelance Developer & Consultant
> > > [Lucene.NET](http://Lucene.NET) committer and PMC member
> > >
> > > On Thu, Feb 5, 2015 at 1:26 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> > >
> > > > Hi Guys,
> > > >
> > > > We use Elasticsearch as our tracking system of our products in a
> > > > dynamic to track performance.
> > > > Searching of this data is being used by small group of users (10-12) in
> > > > the company to measure performance.
> > > >
> > > > In our current environment, we see limit of 15,000 documents being
> > > > inserted per second without the ability to scale.
> > > >
> > > > Some information on the current setup and the flow:
> > > >
> > > > _- Tracking servers_
> > > > 8 x Apache servers behind Amazon ELB which serve empty html files so it
> > > > tracks the parameters given and writes it to Apache access log.
> > > > on each server, we also have Logstash which configured to read this
> > > > access log file and send data to Elasticsearch cluster.
> > > >
> > > > \*- Elasticsearch Cluster: \*
> > > > 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> > > > process - 30GB Heap size
> > > > 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> > > > processes each with 30GB Heap size.
> > > >
> > > > Additional information on the Cluster:
> > > > [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt)
> > > >
> > > > Cluster health: [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#)
> > > > file-gistfile1-txt
> > > >
> > > > Logstash Configuration: [https://gist.github.com/](https://gist.github.com/)
> > > > hagait/23f4b2bc614a4c4acbb6
> > > >
> > > > Elasticsearch configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait/)
> > > > ba3684048abe2f9219b8
> > > >
> > > > Thank you for the support!
> > > > Regards,
> > > > Hagai
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%
> > > > [40googlegroups.com](http://40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 9, 2015, 10:22am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/6 "2015-02-09T10:22:22Z")
Logstash is CPU bound, SSD won't help. It's a JRuby implementation. Try to
see if you can have multiple logstash shippers on the same logs. Having a
redis / kafka server as a middle tier is also a general practice. If that
is not feasible then yes - my advise to you would be to roll your own.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Mon, Feb 9, 2015 at 12:15 PM, Hagai T [hagai.tob@gmail.com](mailto:hagai.tob@gmail.com) wrote:
> Hi,
>
> We were able to identify the bottleneck which seems to be the Logstash
> service.
> It seems that the Elasticsearch cluster is able to handle 40,000 per
> second documents with a 3 ES servers cluster using Java client that was
> written by us using SDK with bulk inserts.
> The client (written for load testing) is generating JSON format and send
> it to Elasticsearch for further processing.
>
> We run the same test with Logstash which reads the JSON format from Apache
> access log on a general purpose SSD and managed to achieve maximum of 4,000
> requests per second.
> With 2 Logstash servers we achieved 8,000 req per second.
>
> Getting rid of the filtering section in logstash configuration file helped
> us get to this number. with filtering we achieved only 1,5000-2,000 req per
> sec.
> I also tried to move the log file to Ephemeral storage but didn't get any
> improvement.
> We don't have any resources problem in the Logstash server (I/O / CPU) so
> it seems like a limit in the file input module or either logstash itself.
>
> I was able to test Logstash performance by creating huge log file 2GB and
> starting Logstash to send it's content.
> I also did tried with smaller files (4-5MB each) but performance didn't
> get any better.
>
> Does it sound reasonable for you guys that I got to a limit of 4,000 req
> per second with one Logstash?
> If you have any suggestions of how to proceed from here I will be more
> than happy to hear that.
>
> If we can't get more from one Logstash, we'll have to develop our own Java
> service to do that instead.
>
> _Apache Access log file output example - (already in JSON Format):_
> { "timestamp":"2015-02-09T10:07:48+0000",
> "bq\_timestamp":"2015-02-09T10:07:48", "client\_ip":"52.2.11.111",
> "client\_port":"80", "latency\_ms":"57", "latency\_sec":"0",
> "elb\_status\_code":"200",
> "request":"/il.html?e=fpAdOpportunity&w=wfl\_dose&vid=1&vname=compName\_PM&ecpm=8&adid=1814157&media\_file\_type=MEDIA\_FILE\_TYPE&media\_file\_url=MEDIA\_FILE\_URL¤t\_url=%0Ahttp%3A%2F%
> [2Fu-sd.gga.tv](http://2Fu-sd.gga.tv)
> %2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau\_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%[252F3ffese.com](http://252F3ffese.com)%26eov%3Deov%0A%09¤t\_main\_vast\_url=MAIN\_VAST\_URL&error\_code=ERROR\_CODE&error\_message=ERROR\_MESSAGE&q9=
> [dsdase.com](http://dsdase.com)&[apid=dose.com](http://apid=dose.com)&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",
> "request\_path":"/il.html", "referer":"-", "user\_agent":"Mozilla/5.0
> (redhat-x86\_64-linux-gnu) Siege/3.0.8" }
>
> _Logstash configuration file (for the testing I ran it with root without
> any limitations):_
> input {
> file {
> path =\>
> "/var/log/httpd/aaa.ddddd.com.logstash-acc.log.[0-9]\*"
> codec =\> json
> type =\> "tracking"
> discover\_interval =\> 1
> sincedb\_path =\> "/opt/logstash/httpd-sincedb"
> sincedb\_write\_interval =\> 1
> }
> }
>
> output {
> elasticsearch {
> workers =\> 1
> host =\> "[aaa.dddd.com](http://aaa.dddd.com)"
> index =\> "%{request\_path}-logstash-%{+YYYY-MM-dd}"
> flush\_size =\> 1000
> cluster =\> "video"
> codec =\> json
> }
> }
>
> Your help is appreciated!
> Thanks!
>
> On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko
> wrote:
>
> > I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use
> > of the snapshot/restore API will provide you HA and DR guarantees.
> >
> > The rejections you are seeing are due to slow I/O operations, because the
> > disk is not local. There is a way to have a bigger queue but I'd advise
> > against that and instead go with a local fast disk.
> >
> > --
> >
> > Itamar Syn-Hershko
> > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > Freelance Developer & Consultant
> > [Lucene.NET](http://Lucene.NET) committer and PMC member
> >
> > On Thu, Feb 5, 2015 at 1:51 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> >
> > > Hi Itamar, thank you for the reply.
> > >
> > > This is 15k inserts totally and not for one host in the cluster.
> > > Yes, we have 15 sharding whiting one index. shards are spreaded on the
> > > nodes equally (automatically by Elasticsearch cluster).
> > > We currently use general purpose SSD and not Ephemeral storage.
> > >
> > > In addition, I see a lot of thread pool bulk rejections from the
> > > Elasticsearch side.
> > >
> > > On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
> > > wrote:
> > >
> > > > What is the question?...
> > > >
> > > > 15k inserts per sec per node is actually quite nice.
> > > >
> > > > Are your index sharded? If you write to one index only, you write to
> > > > maximum of x nodes where x is the number of shards of that index. Since
> > > > shards of the same index can co-exist on one node, check if you are
> > > > spanning writes.
> > > >
> > > > Use local disks - never EBS, and if you really care about writing
> > > > speeds use SSDs.
> > > >
> > > > Other than that, Mike did an excellent write up on the subject :
> > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/performance-consideratio)
> > > > ns-elasticsearch-indexing/
> > > >
> > > > --
> > > >
> > > > Itamar Syn-Hershko
> > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > > > Freelance Developer & Consultant
> > > > [Lucene.NET](http://Lucene.NET) committer and PMC member
> > > >
> > > > On Thu, Feb 5, 2015 at 1:26 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> > > >
> > > > > Hi Guys,
> > > > >
> > > > > We use Elasticsearch as our tracking system of our products in a
> > > > > dynamic to track performance.
> > > > > Searching of this data is being used by small group of users (10-12)
> > > > > in the company to measure performance.
> > > > >
> > > > > In our current environment, we see limit of 15,000 documents being
> > > > > inserted per second without the ability to scale.
> > > > >
> > > > > Some information on the current setup and the flow:
> > > > >
> > > > > _- Tracking servers_
> > > > > 8 x Apache servers behind Amazon ELB which serve empty html files so
> > > > > it tracks the parameters given and writes it to Apache access log.
> > > > > on each server, we also have Logstash which configured to read this
> > > > > access log file and send data to Elasticsearch cluster.
> > > > >
> > > > > \*- Elasticsearch Cluster: \*
> > > > > 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> > > > > process - 30GB Heap size
> > > > > 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> > > > > processes each with 30GB Heap size.
> > > > >
> > > > > Additional information on the Cluster:
> > > > > [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-gistfile1-txt)
> > > > >
> > > > > Cluster health: [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#)
> > > > > file-gistfile1-txt
> > > > >
> > > > > Logstash Configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait)
> > > > > /23f4b2bc614a4c4acbb6
> > > > >
> > > > > Elasticsearch configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait/)
> > > > > ba3684048abe2f9219b8
> > > > >
> > > > > Thank you for the support!
> > > > > Regards,
> > > > > Hagai
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40goo
> > > > > [glegroups.com](http://glegroups.com)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%
> > > > [40googlegroups.com](http://40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > >
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt4iLntP7Te3zX0ophQ7OXZe%2BvzLUyXBbpELNgkv3nVeQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zt4iLntP7Te3zX0ophQ7OXZe%2BvzLUyXBbpELNgkv3nVeQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Hagai\_T](https://discuss.elastic.co/u/Hagai_T)
#### Post date: [February 9, 2015, 10:38am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/7 "2015-02-09T10:38:35Z")
Hi Itamar,
In my case it doesn't seems to be a CPU bottleneck at all.
The instance is actually doing nothing and this is what I don't understand.
I tried to tweak some of the configurations like workers but more than 4
doesn't seem to improve anything.
Maybe I am missing something?
On Monday, February 9, 2015 at 12:22:28 PM UTC+2, Itamar Syn-Hershko wrote:
> Logstash is CPU bound, SSD won't help. It's a JRuby implementation. Try to
> see if you can have multiple logstash shippers on the same logs. Having a
> redis / kafka server as a middle tier is also a general practice. If that
> is not feasible then yes - my advise to you would be to roll your own.
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Mon, Feb 9, 2015 at 12:15 PM, Hagai T \<[haga...@gmail.com](mailto:haga...@gmail.com) \\>
> wrote:
>
> > Hi,
> >
> > We were able to identify the bottleneck which seems to be the Logstash
> > service.
> > It seems that the Elasticsearch cluster is able to handle 40,000 per
> > second documents with a 3 ES servers cluster using Java client that was
> > written by us using SDK with bulk inserts.
> > The client (written for load testing) is generating JSON format and send
> > it to Elasticsearch for further processing.
> >
> > We run the same test with Logstash which reads the JSON format from
> > Apache access log on a general purpose SSD and managed to achieve maximum
> > of 4,000 requests per second.
> > With 2 Logstash servers we achieved 8,000 req per second.
> >
> > Getting rid of the filtering section in logstash configuration file
> > helped us get to this number. with filtering we achieved only 1,5000-2,000
> > req per sec.
> > I also tried to move the log file to Ephemeral storage but didn't get any
> > improvement.
> > We don't have any resources problem in the Logstash server (I/O / CPU) so
> > it seems like a limit in the file input module or either logstash itself.
> >
> > I was able to test Logstash performance by creating huge log file 2GB and
> > starting Logstash to send it's content.
> > I also did tried with smaller files (4-5MB each) but performance didn't
> > get any better.
> >
> > Does it sound reasonable for you guys that I got to a limit of 4,000 req
> > per second with one Logstash?
> > If you have any suggestions of how to proceed from here I will be more
> > than happy to hear that.
> >
> > If we can't get more from one Logstash, we'll have to develop our own
> > Java service to do that instead.
> >
> > _Apache Access log file output example - (already in JSON Format):_
> > { "timestamp":"2015-02-09T10:07:48+0000",
> > "bq\_timestamp":"2015-02-09T10:07:48", "client\_ip":"52.2.11.111",
> > "client\_port":"80", "latency\_ms":"57", "latency\_sec":"0",
> > "elb\_status\_code":"200",
> > "request":"/il.html?e=fpAdOpportunity&w=wfl\_dose&vid=1&vname=compName\_PM&ecpm=8&adid=1814157&media\_file\_type=MEDIA\_FILE\_TYPE&media\_file\_url=MEDIA\_FILE\_URL¤t\_url=%0Ahttp%3A%2F%
> > [2Fu-sd.gga.tv](http://2Fu-sd.gga.tv)
> > %2Fa%2Fh%2FJvf82UX3%2Beff48Z%2fwU20swbapQoWau\_%3Fcb%3D5605126933660359000%26pet%3Dpreroll%26pageUrl%3Dhttp%253A%252F%[252F3ffese.com](http://252F3ffese.com)%26eov%3Deov%0A%09¤t\_main\_vast\_url=MAIN\_VAST\_URL&error\_code=ERROR\_CODE&error\_message=ERROR\_MESSAGE&q9=
> > [dsdase.com](http://dsdase.com)&[apid=dose.com](http://apid=dose.com)&d=Convert&device=6719&csize=300X250&token=14123669&cb=260174713417&pc=PLAYCOUNT",
> > "request\_path":"/il.html", "referer":"-", "user\_agent":"Mozilla/5.0
> > (redhat-x86\_64-linux-gnu) Siege/3.0.8" }
> >
> > _Logstash configuration file (for the testing I ran it with root without
> > any limitations):_
> > input {
> > file {
> > path =\>
> > "/var/log/httpd/aaa.ddddd.com.logstash-acc.log.[0-9]\*"
> > codec =\> json
> > type =\> "tracking"
> > discover\_interval =\> 1
> > sincedb\_path =\> "/opt/logstash/httpd-sincedb"
> > sincedb\_write\_interval =\> 1
> > }
> > }
> >
> > output {
> > elasticsearch {
> > workers =\> 1
> > host =\> "[aaa.dddd.com](http://aaa.dddd.com)"
> > index =\> "%{request\_path}-logstash-%{+YYYY-MM-dd}"
> > flush\_size =\> 1000
> > cluster =\> "video"
> > codec =\> json
> > }
> > }
> >
> > Your help is appreciated!
> > Thanks!
> >
> > On Thursday, February 5, 2015 at 1:56:34 PM UTC+2, Itamar Syn-Hershko
> > wrote:
> >
> > > I'd recommend you use ephemeral SSD - 2+ factor replicas and proper use
> > > of the snapshot/restore API will provide you HA and DR guarantees.
> > >
> > > The rejections you are seeing are due to slow I/O operations, because
> > > the disk is not local. There is a way to have a bigger queue but I'd advise
> > > against that and instead go with a local fast disk.
> > >
> > > --
> > >
> > > Itamar Syn-Hershko
> > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > > Freelance Developer & Consultant
> > > [Lucene.NET](http://Lucene.NET) committer and PMC member
> > >
> > > On Thu, Feb 5, 2015 at 1:51 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> > >
> > > > Hi Itamar, thank you for the reply.
> > > >
> > > > This is 15k inserts totally and not for one host in the cluster.
> > > > Yes, we have 15 sharding whiting one index. shards are spreaded on the
> > > > nodes equally (automatically by Elasticsearch cluster).
> > > > We currently use general purpose SSD and not Ephemeral storage.
> > > >
> > > > In addition, I see a lot of thread pool bulk rejections from the
> > > > Elasticsearch side.
> > > >
> > > > On Thursday, February 5, 2015 at 1:33:05 PM UTC+2, Itamar Syn-Hershko
> > > > wrote:
> > > >
> > > > > What is the question?...
> > > > >
> > > > > 15k inserts per sec per node is actually quite nice.
> > > > >
> > > > > Are your index sharded? If you write to one index only, you write to
> > > > > maximum of x nodes where x is the number of shards of that index. Since
> > > > > shards of the same index can co-exist on one node, check if you are
> > > > > spanning writes.
> > > > >
> > > > > Use local disks - never EBS, and if you really care about writing
> > > > > speeds use SSDs.
> > > > >
> > > > > Other than that, Mike did an excellent write up on the subject :
> > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/performance-consideratio)
> > > > > ns-elasticsearch-indexing/
> > > > >
> > > > > --
> > > > >
> > > > > Itamar Syn-Hershko
> > > > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > > > > Freelance Developer & Consultant
> > > > > [Lucene.NET](http://Lucene.NET) committer and PMC member
> > > > >
> > > > > On Thu, Feb 5, 2015 at 1:26 PM, Hagai T [haga...@gmail.com](mailto:haga...@gmail.com) wrote:
> > > > >
> > > > > > Hi Guys,
> > > > > >
> > > > > > We use Elasticsearch as our tracking system of our products in a
> > > > > > dynamic to track performance.
> > > > > > Searching of this data is being used by small group of users (10-12)
> > > > > > in the company to measure performance.
> > > > > >
> > > > > > In our current environment, we see limit of 15,000 documents being
> > > > > > inserted per second without the ability to scale.
> > > > > >
> > > > > > Some information on the current setup and the flow:
> > > > > >
> > > > > > _- Tracking servers_
> > > > > > 8 x Apache servers behind Amazon ELB which serve empty html files so
> > > > > > it tracks the parameters given and writes it to Apache access log.
> > > > > > on each server, we also have Logstash which configured to read this
> > > > > > access log file and send data to Elasticsearch cluster.
> > > > > >
> > > > > > \*- Elasticsearch Cluster: \*
> > > > > > 4 x r3.2xlarge (61.0 GB RAM, 8 cores) - contains one Elasticsearch
> > > > > > process - 30GB Heap size
> > > > > > 1 x r3.4xlarge (122.0 GB RAM, 16 cores) - contains two Elasticsearch
> > > > > > processes each with 30GB Heap size.
> > > > > >
> > > > > > Additional information on the Cluster:
> > > > > > [Additional information · GitHub](https://gist.github.com/hagait/61e3fac181ff413a8b8c#file-)
> > > > > > gistfile1-txt
> > > > > >
> > > > > > Cluster health: [Cluster Health · GitHub](https://gist.github.com/hagait/d4f16d8f7b724f85b0ee#)
> > > > > > file-gistfile1-txt
> > > > > >
> > > > > > Logstash Configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait)
> > > > > > /23f4b2bc614a4c4acbb6
> > > > > >
> > > > > > Elasticsearch configuration: [hagait’s gists · GitHub](https://gist.github.com/hagait/)
> > > > > > ba3684048abe2f9219b8
> > > > > >
> > > > > > Thank you for the support!
> > > > > > Regards,
> > > > > > Hagai
> > > > > >
> > > > > > --
> > > > > > You received this message because you are subscribed to the Google
> > > > > > Groups "elasticsearch" group.
> > > > > > To unsubscribe from this group and stop receiving emails from it,
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > > msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40goo
> > > > > > [glegroups.com](http://glegroups.com)
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9c5352ed-010d-4080-9794-b8dc0c2a5370%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > > .
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > >
> > > > > --
> > > > > You received this message because you are subscribed to the Google
> > > > > Groups "elasticsearch" group.
> > > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > > msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%
> > > > > [40googlegroups.com](http://40googlegroups.com)
> > > > > [https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/318b46ed-626f-4975-a417-c99ada9c30fe%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > > .
> > > >
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/75c6a8db-87ab-432b-9974-55cfc44f139f%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b571469f-6725-48f6-ae63-ce8918599d22%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b571469f-6725-48f6-ae63-ce8918599d22%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:34am UTC](https://discuss.elastic.co/t/performance-limitation-with-elk-stack/21990/8 "2017-07-06T00:34:13Z")