ElasticSearch for higher frequency time series data

Elastic Stack Elasticsearch
1

1> Let's assume the time series data comes from distributed nodes like we
want to collect CPU counter on 100's of servers in the DC at high
frequency(4 times a sec). Does anybody have any experience using
ElasticSearch as the time series data storage and query engine? Any gotcha
doing so?

2> Also, is logstash the way to collect and send the data at each host to
ES?

Thank you,

Mike

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Any reason for ES and not something like graphite?

ES should be able to do it though.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 April 2014 15:30, mikesam460@gmail.com wrote:

1> Let's assume the time series data comes from distributed nodes like we
want to collect CPU counter on 100's of servers in the DC at high
frequency(4 times a sec). Does anybody have any experience using
Elasticsearch as the time series data storage and query engine? Any gotcha
doing so?

2> Also, is logstash the way to collect and send the data at each host to
ES?

Thank you,

Mike

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zn1XTG3MHBk72Oq4xa_j9q0qkRDNYD0YRktL0%3Dy%3DRt8A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Well, we are thinking to store both logs and metrics in one place. In
theory it could work but I am wondering if anybody has actually tired that?

On Tue, Apr 8, 2014 at 10:37 PM, Mark Walkom markw@campaignmonitor.comwrote:

Any reason for ES and not something like graphite?

ES should be able to do it though.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 April 2014 15:30, mikesam460@gmail.com wrote:

1> Let's assume the time series data comes from distributed nodes like we
want to collect CPU counter on 100's of servers in the DC at high
frequency(4 times a sec). Does anybody have any experience using
Elasticsearch as the time series data storage and query engine? Any gotcha
doing so?

2> Also, is logstash the way to collect and send the data at each host to
ES?

Thank you,

Mike

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qkPiqXFTlXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624Zn1XTG3MHBk72Oq4xa_j9q0qkRDNYD0YRktL0%3Dy%3DRt8A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624Zn1XTG3MHBk72Oq4xa_j9q0qkRDNYD0YRktL0%3Dy%3DRt8A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Thanks,
Mike

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CACiRYOef%2BSf1khEowoHMTXXNzkqFxfnmhzERXEeL%2BAq9-O0S1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

I gave it a test a while ago and it seemed to work ok (using collectd +
logstash), that was only with one server though and not at that resolution.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 April 2014 15:48, Mike Sam mikesam460@gmail.com wrote:

Well, we are thinking to store both logs and metrics in one place. In
theory it could work but I am wondering if anybody has actually tired that?

On Tue, Apr 8, 2014 at 10:37 PM, Mark Walkom markw@campaignmonitor.comwrote:

Any reason for ES and not something like graphite?

ES should be able to do it though.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 April 2014 15:30, mikesam460@gmail.com wrote:

1> Let's assume the time series data comes from distributed nodes like
we want to collect CPU counter on 100's of servers in the DC at high
frequency(4 times a sec). Does anybody have any experience using
Elasticsearch as the time series data storage and query engine? Any gotcha
doing so?

2> Also, is logstash the way to collect and send the data at each host
to ES?

Thank you,

Mike

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/0bc154cb-f0ad-4480-9177-8f29adb3da4c%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qkPiqXFTlXw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624Zn1XTG3MHBk72Oq4xa_j9q0qkRDNYD0YRktL0%3Dy%3DRt8A%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624Zn1XTG3MHBk72Oq4xa_j9q0qkRDNYD0YRktL0%3Dy%3DRt8A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
Thanks,
Mike

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CACiRYOef%2BSf1khEowoHMTXXNzkqFxfnmhzERXEeL%2BAq9-O0S1w%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CACiRYOef%2BSf1khEowoHMTXXNzkqFxfnmhzERXEeL%2BAq9-O0S1w%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Y3XGkLcCeOw5dCqdMda%3DWxAT5h%3DSXSSaa_L2QvzXvz2Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.