# How to measure the performance of ELK system? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 29, 2015, 3:19am UTC](https://discuss.elastic.co/t/how-to-measure-the-performance-of-elk-system/21891 "2015-01-29T03:19:31Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Yiming\_Li](https://avatars.discourse-cdn.com/v4/letter/y/54ee81/32.png) [@Yiming\_Li](https://discuss.elastic.co/u/Yiming_Li)\ **Post date:** [January 29, 2015, 3:19am UTC](https://discuss.elastic.co/t/how-to-measure-the-performance-of-elk-system/21891/1 "2015-01-29T03:19:31Z")
Dear all, I just started to learn and experiment with ELK in a logging related project at work. The answer I am looking for is that: say I am shipping tons of data into logstash, do some basic transformation, and feed the data into elasticsearch, finally view dashboard in Kibana. How long this whole process take under load pressure? I know this is a very general problem statement and the answer is like to vary depending on machines, network etc. But I want to find out a generic way to measure this, then I can tweak different machine/network configurations. What I have in mind is that: 1. record the timestamp when generating the log entry, say t1 2. add another timestamp in logstash, say t2 3. somehow figure out a third timestamp t3, which is the time when the log entry is available in Kibana. Then I will monitor the difference among t1, t2, and t3 in Kibana(assuming the time on all machines are in sync), while scaling up the log(load) generator. Does this sound like a viable approach? if so, can you help me with some instructions on how to get t2 and t3? (I have been struggling with this step) Thank you very much. Yiming -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 29, 2015, 10:00pm UTC](https://discuss.elastic.co/t/how-to-measure-the-performance-of-elk-system/21891/2 "2015-01-29T22:00:30Z")
That seems like a sane idea, however adding the 3rd timestamp might be hard. It'd probably be easier to let ES generate @timestamp and then leverage that as your end point timestamp, as KB is simply pulling the data from ES. On 29 January 2015 at 14:19, Yiming Li [nicneo925@gmail.com](mailto:nicneo925@gmail.com) wrote: > Dear all, > > I just started to learn and experiment with ELK in a logging related > project at work. > > The answer I am looking for is that: say I am shipping tons of data into > logstash, do some basic transformation, and feed the data into > elasticsearch, finally view dashboard in Kibana. How long this whole > process take under load pressure? > > I know this is a very general problem statement and the answer is like to > vary depending on machines, network etc. But I want to find out a generic > way to measure this, then I can tweak different machine/network > configurations. > > What I have in mind is that: > > 1. record the timestamp when generating the log entry, say t1 > 2. add another timestamp in logstash, say t2 > 3. somehow figure out a third timestamp t3, which is the time when the log > entry is available in Kibana. > > Then I will monitor the difference among t1, t2, and t3 in Kibana(assuming > the time on all machines are in sync), while scaling up the log(load) > generator. > > Does this sound like a viable approach? if so, can you help me with some > instructions on how to get t2 and t3? (I have been struggling with this > step) > > Thank you very much. > > Yiming > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9N1oV-LD\_sZOtfy538uy7VQBC4GhnEChLF3U5LRN-4QA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9N1oV-LD_sZOtfy538uy7VQBC4GhnEChLF3U5LRN-4QA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Yiming\_Li](https://avatars.discourse-cdn.com/v4/letter/y/54ee81/32.png) [@Yiming\_Li](https://discuss.elastic.co/u/Yiming_Li)\ **Post date:** [January 29, 2015, 11:22pm UTC](https://discuss.elastic.co/t/how-to-measure-the-performance-of-elk-system/21891/3 "2015-01-29T23:22:07Z")
Hi Mark, Thanks for the reply. I will definitely try adding timestamp in the mapping, as discussed here: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-timestamp-field.html). It seems that logstash will also generate a default @timestamp, if there is no timestamp in the log itself, I will leverage that a too. I will also look at the bigdesk plugin to identify Elasticsearch system to see at what moment resources are exhausted. Hopefully this combination can give us some interesting result. Yiming On Thursday, January 29, 2015 at 2:01:05 PM UTC-8, Mark Walkom wrote: > That seems like a sane idea, however adding the 3rd timestamp might be > hard. > It'd probably be easier to let ES generate @timestamp and then leverage > that as your end point timestamp, as KB is simply pulling the data from ES. > > On 29 January 2015 at 14:19, Yiming Li \<[nicn...@gmail.com](mailto:nicn...@gmail.com) \\> > wrote: > > > Dear all, > > > > I just started to learn and experiment with ELK in a logging related > > project at work. > > > > The answer I am looking for is that: say I am shipping tons of data into > > logstash, do some basic transformation, and feed the data into > > elasticsearch, finally view dashboard in Kibana. How long this whole > > process take under load pressure? > > > > I know this is a very general problem statement and the answer is like to > > vary depending on machines, network etc. But I want to find out a generic > > way to measure this, then I can tweak different machine/network > > configurations. > > > > What I have in mind is that: > > > > 1. record the timestamp when generating the log entry, say t1 > > 2. add another timestamp in logstash, say t2 > > 3. somehow figure out a third timestamp t3, which is the time when the > > log entry is available in Kibana. > > > > Then I will monitor the difference among t1, t2, and t3 in > > Kibana(assuming the time on all machines are in sync), while scaling up the > > log(load) generator. > > > > Does this sound like a viable approach? if so, can you help me with some > > instructions on how to get t2 and t3? (I have been struggling with this > > step) > > > > Thank you very much. > > > > Yiming > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/00663a14-835d-4724-b9c4-2b4c0a36d147%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b8531c77-5a60-4114-b243-1398f7bec2ee%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b8531c77-5a60-4114-b243-1398f7bec2ee%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:35am UTC](https://discuss.elastic.co/t/how-to-measure-the-performance-of-elk-system/21891/4 "2017-07-06T00:35:47Z")