# Need help/suggestion for the massive queries user case

**URL:** <https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567>\
**Category:** Elasticsearch\
**Created:** [September 2, 2014, 6:53am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567 "2014-09-02T06:53:09Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 6:53am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/1 "2014-09-02T06:53:09Z")

</div>

Hi guys,

I have some streaming sensor data as input to ES. For each incoming data  
message, I need to do a query on the historic data in ES according to the  
'timestamp' and 'messageId' in that message. I need to get the aggregated  
query results in real-time.

My problem is each data message may sprout 28 queries at approximately the  
same time and I get [No-nodes available] error from ES frequently.

Can anyone suggest what is the cause of this error and how can I fix it? Or  
should I use some other system to handle this problem like apache storm?

Thanks!

Yuheng

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 7:09am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/2 "2014-09-02T07:09:55Z")

</div>

May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
The main cost is of course in term of machine and disk space.

Can't tell really more without more details about what you are doing exactly. May be others have some ideas?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com)) a écrit:

Hi guys,

I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.

My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.

Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?

Thanks!

## Yuheng

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.54056d43.741226bb.6ef9%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54056d43.741226bb.6ef9%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 7:17am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/3 "2014-09-02T07:17:23Z")

</div>

Each query is doing a search in \_all indexes in the cluster. Maybe that's  
also a hold up of the availability of ES server.

I tried to run 3 ES nodes on my desktop machine but it does not seems to  
help, it also crashed after some time.

On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:

> May be you could increase the number of replicas and nodes so your cluster  
> could handle more queries at a time.  
> The main cost is of course in term of machine and disk space.
> 
> Can't tell really more without more details about what you are doing  
> exactly. May be others have some ideas?
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)  
> \<javascript:\>) a écrit:
> 
> Hi guys,
> 
> I have some streaming sensor data as input to ES. For each incoming data  
> message, I need to do a query on the historic data in ES according to the  
> 'timestamp' and 'messageId' in that message. I need to get the aggregated  
> query results in real-time.
> 
> My problem is each data message may sprout 28 queries at approximately the  
> same time and I get [No-nodes available] error from ES frequently.
> 
> Can anyone suggest what is the cause of this error and how can I fix it?  
> Or should I use some other system to handle this problem like apache storm?
> 
> Thanks!
> 
> ## Yuheng
> 
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 7:46am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/4 "2014-09-02T07:46:12Z")

</div>

Ha! On your desktop? So that's not a production issue.

In that case, you should may be give more memory to all your elasticsearch nodes.  
What is your current HEAP size?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com)) a écrit:

Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.

I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.

On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
The main cost is of course in term of machine and disk space.

Can't tell really more without more details about what you are doing exactly. May be others have some ideas?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

Hi guys,

I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.

My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.

Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?

Thanks!

## Yuheng

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.540575c5.3f6ab60f.6ef9%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.540575c5.3f6ab60f.6ef9%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 1:49pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/5 "2014-09-02T13:49:37Z")

</div>

I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,

How can I deal with this?

Thanks

On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:

> Ha! On your desktop? So that's not a production issue.
> 
> In that case, you should may be give more memory to all your elasticsearch  
> nodes.  
> What is your current HEAP size?
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)  
> \<javascript:\>) a écrit:
> 
> Each query is doing a search in \_all indexes in the cluster. Maybe that's  
> also a hold up of the availability of ES server.
> 
> I tried to run 3 ES nodes on my desktop machine but it does not seems to  
> help, it also crashed after some time.
> 
> On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> 
> > May be you could increase the number of replicas and nodes so your  
> > cluster could handle more queries at a time.  
> > The main cost is of course in term of machine and disk space.
> > 
> > Can't tell really more without more details about what you are doing  
> > exactly. May be others have some ideas?
> > 
> > ```
> > -- 
> > 
> > ```
> > 
> > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > 
> > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > 
> > Hi guys,
> > 
> > I have some streaming sensor data as input to ES. For each incoming data  
> > message, I need to do a query on the historic data in ES according to the  
> > 'timestamp' and 'messageId' in that message. I need to get the aggregated  
> > query results in real-time.
> > 
> > My problem is each data message may sprout 28 queries at approximately  
> > the same time and I get [No-nodes available] error from ES frequently.
> > 
> > Can anyone suggest what is the cause of this error and how can I fix it?  
> > Or should I use some other system to handle this problem like apache storm?
> > 
> > Thanks!
> > 
> > ## Yuheng
> > 
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 2:04pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/6 "2014-09-02T14:04:19Z")

</div>

Try 2g

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

> Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com) a écrit :
> 
> I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> 
> How can I deal with this?
> 
> Thanks
> 
> > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:  
> > Ha! On your desktop? So that's not a production issue.
> > 
> > In that case, you should may be give more memory to all your elasticsearch nodes.  
> > What is your current HEAP size?
> > 
> > --  
> > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > @dadoonet | @elasticsearchfr
> > 
> > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > 
> > > Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.
> > > 
> > > I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.
> > > 
> > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
> > > > May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
> > > > The main cost is of course in term of machine and disk space.
> > > > 
> > > > Can't tell really more without more details about what you are doing exactly. May be others have some ideas?
> > > > 
> > > > --  
> > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > > > @dadoonet | @elasticsearchfr
> > > > 
> > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > > > 
> > > > > Hi guys,
> > > > > 
> > > > > I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.
> > > > > 
> > > > > My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.
> > > > > 
> > > > > Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?
> > > > > 
> > > > > Thanks!
> > > > > 
> > > > > ## Yuheng
> > > > > 
> > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com).  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > --  
> > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com).  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/E0618783-8E8E-4D42-8A9D-AE747509C1B4%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/E0618783-8E8E-4D42-8A9D-AE747509C1B4%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 2:14pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/7 "2014-09-02T14:14:28Z")

</div>

I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
Currently some of my shards are stated as "unassigned" in my kopf web  
interface. It happened after I started 3 nodes at the same time in my  
desktop yesterday.

There should be 3G of data in the cluster, but know I see only 1.92G. I  
think some data is stored in another node in my machine but I don't know  
how to fix it.

[https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
How can I fix this issue?

On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:

> Try 2g
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 2 sept. 2014 à 15:49, Yuheng Du \<[yuheng....@gmail.com](mailto:yuheng....@gmail.com) \<javascript:\>\> a  
> écrit :
> 
> I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> 
> How can I deal with this?
> 
> Thanks
> 
> On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> 
> > Ha! On your desktop? So that's not a production issue.
> > 
> > In that case, you should may be give more memory to all your  
> > elasticsearch nodes.  
> > What is your current HEAP size?
> > 
> > --  
> > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > 
> > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > 
> > Each query is doing a search in \_all indexes in the cluster. Maybe that's  
> > also a hold up of the availability of ES server.
> > 
> > I tried to run 3 ES nodes on my desktop machine but it does not seems to  
> > help, it also crashed after some time.
> > 
> > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > 
> > > May be you could increase the number of replicas and nodes so your  
> > > cluster could handle more queries at a time.  
> > > The main cost is of course in term of machine and disk space.
> > > 
> > > Can't tell really more without more details about what you are doing  
> > > exactly. May be others have some ideas?
> > > 
> > > ```
> > > -- 
> > > 
> > > ```
> > > 
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > 
> > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > écrit:
> > > 
> > > Hi guys,
> > > 
> > > I have some streaming sensor data as input to ES. For each incoming data  
> > > message, I need to do a query on the historic data in ES according to the  
> > > 'timestamp' and 'messageId' in that message. I need to get the aggregated  
> > > query results in real-time.
> > > 
> > > My problem is each data message may sprout 28 queries at approximately  
> > > the same time and I get [No-nodes available] error from ES frequently.
> > > 
> > > Can anyone suggest what is the cause of this error and how can I fix it?  
> > > Or should I use some other system to handle this problem like apache storm?
> > > 
> > > Thanks!
> > > 
> > > ## Yuheng
> > > 
> > > You received this message because you are subscribed to the Google  
> > > Groups "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send  
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 2:26pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/8 "2014-09-02T14:26:09Z")

</div>

You have to start again the second node on the same machine to get the other shards back. Then when you cluster is yellow again, you can switch off the second node.

BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this will do the same thing behind the scene). See [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#\_environment\_variables](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com)) a écrit:

I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g. Currently some of my shards are stated as "unassigned" in my kopf web interface. It happened after I started 3 nodes at the same time in my desktop yesterday.

There should be 3G of data in the cluster, but know I see only 1.92G. I think some data is stored in another node in my machine but I don't know how to fix it.

How can I fix this issue?

On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:  
Try 2g

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :

I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,

How can I deal with this?

Thanks

On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:  
Ha! On your desktop? So that's not a production issue.

In that case, you should may be give more memory to all your elasticsearch nodes.  
What is your current HEAP size?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.

I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.

On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
The main cost is of course in term of machine and disk space.

Can't tell really more without more details about what you are doing exactly. May be others have some ideas?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

Hi guys,

I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.

My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.

Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?

Thanks!

## Yuheng

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.5405d381.515f007c.d74c%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.5405d381.515f007c.d74c%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 6:31pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/9 "2014-09-02T18:31:28Z")

</div>

Ok. The shards are recovered now. I tried to set the ES\_HEAP\_SIZE to 2g,  
but I am still getting "Nodes not available" error:

[https://lh3.googleusercontent.com/-g8vS5TTJwP0/VAYM0Q0LbLI/AAAAAAAAAOU/zznAKLbY-z0/s1600/nonodes.jpg](https://lh3.googleusercontent.com/-g8vS5TTJwP0/VAYM0Q0LbLI/AAAAAAAAAOU/zznAKLbY-z0/s1600/nonodes.jpg)  
What can be the cause then?

Thanks!

Yuheng

On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:

> You have to start again the second node on the same machine to get the  
> other shards back. Then when you cluster is yellow again, you can switch  
> off the second node.
> 
> BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this  
> will do the same thing behind the scene). See  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)  
> \<javascript:\>) a écrit:
> 
> I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
> Currently some of my shards are stated as "unassigned" in my kopf web  
> interface. It happened after I started 3 nodes at the same time in my  
> desktop yesterday.
> 
> There should be 3G of data in the cluster, but know I see only 1.92G. I  
> think some data is stored in another node in my machine but I don't know  
> how to fix it.
> 
> [https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
> How can I fix this issue?
> 
> On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:
> 
> > Try 2g
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > 
> > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > 
> > How can I deal with this?
> > 
> > Thanks
> > 
> > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> > 
> > > Ha! On your desktop? So that's not a production issue.
> > > 
> > > In that case, you should may be give more memory to all your  
> > > elasticsearch nodes.  
> > > What is your current HEAP size?
> > > 
> > > ```
> > > -- 
> > > 
> > > ```
> > > 
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > 
> > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > écrit:
> > > 
> > > Each query is doing a search in \_all indexes in the cluster. Maybe  
> > > that's also a hold up of the availability of ES server.
> > > 
> > > I tried to run 3 ES nodes on my desktop machine but it does not seems to  
> > > help, it also crashed after some time.
> > > 
> > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > > 
> > > > May be you could increase the number of replicas and nodes so your  
> > > > cluster could handle more queries at a time.  
> > > > The main cost is of course in term of machine and disk space.
> > > > 
> > > > Can't tell really more without more details about what you are doing  
> > > > exactly. May be others have some ideas?
> > > > 
> > > > ```
> > > > -- 
> > > > 
> > > > ```
> > > > 
> > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > 
> > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > écrit:
> > > > 
> > > > Hi guys,
> > > > 
> > > > I have some streaming sensor data as input to ES. For each incoming  
> > > > data message, I need to do a query on the historic data in ES according to  
> > > > the 'timestamp' and 'messageId' in that message. I need to get the  
> > > > aggregated query results in real-time.
> > > > 
> > > > My problem is each data message may sprout 28 queries at approximately  
> > > > the same time and I get [No-nodes available] error from ES frequently.
> > > > 
> > > > Can anyone suggest what is the cause of this error and how can I fix  
> > > > it? Or should I use some other system to handle this problem like apache  
> > > > storm?
> > > > 
> > > > Thanks!
> > > > 
> > > > ## Yuheng
> > > > 
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ff5e5463-cf57-40b1-92ed-1405e3fb0402%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ff5e5463-cf57-40b1-92ed-1405e3fb0402%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 6:37pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/10 "2014-09-02T18:37:54Z")

</div>

BTW, the node stats info shows the jvm usage info as follows:  
"jvm" : {  
"timestamp" : 1409682903048,  
"uptime\_in\_millis" : 488996,  
"mem" : {  
"heap\_used\_in\_bytes" : 48476472,  
"heap\_used\_percent" : 9,  
"heap\_committed\_in\_bytes" : 73412608,  
"heap\_max\_in\_bytes" : 515571712,  
"non\_heap\_used\_in\_bytes" : 57043440,  
"non\_heap\_committed\_in\_bytes" : 92200960,  
"pools" : {  
"young" : {  
"used\_in\_bytes" : 12126760,  
"max\_in\_bytes" : 69795840,  
"peak\_used\_in\_bytes" : 17367040,  
"peak\_max\_in\_bytes" : 69795840  
},  
"survivor" : {  
"used\_in\_bytes" : 915456,  
"max\_in\_bytes" : 8716288,  
"peak\_used\_in\_bytes" : 2097152,  
"peak\_max\_in\_bytes" : 8716288  
},  
"old" : {  
"used\_in\_bytes" : 35434256,  
"max\_in\_bytes" : 437059584,  
"peak\_used\_in\_bytes" : 38372208,  
"peak\_max\_in\_bytes" : 437059584  
}  
}  
},

On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:

> You have to start again the second node on the same machine to get the  
> other shards back. Then when you cluster is yellow again, you can switch  
> off the second node.
> 
> BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this  
> will do the same thing behind the scene). See  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)  
> \<javascript:\>) a écrit:
> 
> I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
> Currently some of my shards are stated as "unassigned" in my kopf web  
> interface. It happened after I started 3 nodes at the same time in my  
> desktop yesterday.
> 
> There should be 3G of data in the cluster, but know I see only 1.92G. I  
> think some data is stored in another node in my machine but I don't know  
> how to fix it.
> 
> [https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
> How can I fix this issue?
> 
> On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:
> 
> > Try 2g
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > 
> > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > 
> > How can I deal with this?
> > 
> > Thanks
> > 
> > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> > 
> > > Ha! On your desktop? So that's not a production issue.
> > > 
> > > In that case, you should may be give more memory to all your  
> > > elasticsearch nodes.  
> > > What is your current HEAP size?
> > > 
> > > ```
> > > -- 
> > > 
> > > ```
> > > 
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > 
> > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > écrit:
> > > 
> > > Each query is doing a search in \_all indexes in the cluster. Maybe  
> > > that's also a hold up of the availability of ES server.
> > > 
> > > I tried to run 3 ES nodes on my desktop machine but it does not seems to  
> > > help, it also crashed after some time.
> > > 
> > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > > 
> > > > May be you could increase the number of replicas and nodes so your  
> > > > cluster could handle more queries at a time.  
> > > > The main cost is of course in term of machine and disk space.
> > > > 
> > > > Can't tell really more without more details about what you are doing  
> > > > exactly. May be others have some ideas?
> > > > 
> > > > ```
> > > > -- 
> > > > 
> > > > ```
> > > > 
> > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > 
> > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > écrit:
> > > > 
> > > > Hi guys,
> > > > 
> > > > I have some streaming sensor data as input to ES. For each incoming  
> > > > data message, I need to do a query on the historic data in ES according to  
> > > > the 'timestamp' and 'messageId' in that message. I need to get the  
> > > > aggregated query results in real-time.
> > > > 
> > > > My problem is each data message may sprout 28 queries at approximately  
> > > > the same time and I get [No-nodes available] error from ES frequently.
> > > > 
> > > > Can anyone suggest what is the cause of this error and how can I fix  
> > > > it? Or should I use some other system to handle this problem like apache  
> > > > storm?
> > > > 
> > > > Thanks!
> > > > 
> > > > ## Yuheng
> > > > 
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 6:48pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/11 "2014-09-02T18:48:08Z")

</div>

I think your JVM settings are incorrect as we can see with heap\_max

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 20:37, Yuheng Du [yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com) a écrit :

BTW, the node stats info shows the jvm usage info as follows:  
"jvm" : {  
"timestamp" : 1409682903048,  
"uptime\_in\_millis" : 488996,  
"mem" : {  
"heap\_used\_in\_bytes" : 48476472,  
"heap\_used\_percent" : 9,  
"heap\_committed\_in\_bytes" : 73412608,  
"heap\_max\_in\_bytes" : 515571712,  
"non\_heap\_used\_in\_bytes" : 57043440,  
"non\_heap\_committed\_in\_bytes" : 92200960,  
"pools" : {  
"young" : {  
"used\_in\_bytes" : 12126760,  
"max\_in\_bytes" : 69795840,  
"peak\_used\_in\_bytes" : 17367040,  
"peak\_max\_in\_bytes" : 69795840  
},  
"survivor" : {  
"used\_in\_bytes" : 915456,  
"max\_in\_bytes" : 8716288,  
"peak\_used\_in\_bytes" : 2097152,  
"peak\_max\_in\_bytes" : 8716288  
},  
"old" : {  
"used\_in\_bytes" : 35434256,  
"max\_in\_bytes" : 437059584,  
"peak\_used\_in\_bytes" : 38372208,  
"peak\_max\_in\_bytes" : 437059584  
}  
}  
},

> On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:  
> You have to start again the second node on the same machine to get the other shards back. Then when you cluster is yellow again, you can switch off the second node.
> 
> BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this will do the same thing behind the scene). See [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> 
> --  
> David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> @dadoonet | @elasticsearchfr
> 
> > Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > 
> > I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g. Currently some of my shards are stated as "unassigned" in my kopf web interface. It happened after I started 3 nodes at the same time in my desktop yesterday.
> > 
> > There should be 3G of data in the cluster, but know I see only 1.92G. I think some data is stored in another node in my machine but I don't know how to fix it.
> > 
> > How can I fix this issue?
> > 
> > > On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:  
> > > Try 2g
> > > 
> > > --  
> > > David 😉  
> > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > 
> > > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > 
> > > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > > > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > > > 
> > > > How can I deal with this?
> > > > 
> > > > Thanks
> > > > 
> > > > > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:  
> > > > > Ha! On your desktop? So that's not a production issue.
> > > > > 
> > > > > In that case, you should may be give more memory to all your elasticsearch nodes.  
> > > > > What is your current HEAP size?
> > > > > 
> > > > > --  
> > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > > > > @dadoonet | @elasticsearchfr
> > > > > 
> > > > > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > > > > 
> > > > > > Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.
> > > > > > 
> > > > > > I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.
> > > > > > 
> > > > > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
> > > > > > > May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
> > > > > > > The main cost is of course in term of machine and disk space.
> > > > > > > 
> > > > > > > Can't tell really more without more details about what you are doing exactly. May be others have some ideas?
> > > > > > > 
> > > > > > > --  
> > > > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > > @dadoonet | @elasticsearchfr
> > > > > > > 
> > > > > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > > > > > > 
> > > > > > > > Hi guys,
> > > > > > > > 
> > > > > > > > I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.
> > > > > > > > 
> > > > > > > > My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.
> > > > > > > > 
> > > > > > > > Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?
> > > > > > > > 
> > > > > > > > Thanks!
> > > > > > > > 
> > > > > > > > ## Yuheng
> > > > > > > > 
> > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com).  
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com).  
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com).  
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com).  
> > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/C7DDDD37-5E0D-4AFD-BB11-490F95879C68%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/C7DDDD37-5E0D-4AFD-BB11-490F95879C68%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 7:05pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/12 "2014-09-02T19:05:51Z")

</div>

Ok. you are right. I was able to set the maximum heap size and now it looks  
like:

"jvm" : {

"timestamp" : 1409684394790,

"uptime\_in\_millis" : 128401,

"mem" : {

"heap\_used\_in\_bytes" : 520385872,

"heap\_used\_percent" : 24,

"heap\_committed\_in\_bytes" : 2138767360,

"heap\_max\_in\_bytes" : 2138767360,

"non\_heap\_used\_in\_bytes" : 46357584,

"non\_heap\_committed\_in\_bytes" : 46727168,

"pools" : {

"young" : {

"used\_in\_bytes" : 61936096,

"max\_in\_bytes" : 69795840,

"peak\_used\_in\_bytes" : 69795840,

"peak\_max\_in\_bytes" : 69795840

},

"survivor" : {

"used\_in\_bytes" : 8716288,

"max\_in\_bytes" : 8716288,

"peak\_used\_in\_bytes" : 8716288,

"peak\_max\_in\_bytes" : 8716288

},

"old" : {

"used\_in\_bytes" : 449733488,

"max\_in\_bytes" : 2060255232,

"peak\_used\_in\_bytes" : 449733488,

"peak\_max\_in\_bytes" : 2060255232

}

}

},

But when I start my two streaming data source which does the query, it  
simply get the ES instance killed. When I restart ES, I was told that:

OpenJDK 64-Bit Server VM warning: INFO:  
os::commit\_memory(0x000000077ad30000, 2060255232, 0) failed; error='Cannot  
allocate memory' (errno=12)

# 

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 2060255232 bytes for

committing reserved memory.

# An error report file with more information is saved as:

# /home/yuheng/Downloads/elasticsearch-1.2.0/bin/hs\_err\_pid5116.log

Any thoughts?

On Tuesday, September 2, 2014 2:48:17 PM UTC-4, David Pilato wrote:

> I think your JVM settings are incorrect as we can see with heap\_max
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 2 sept. 2014 à 20:37, Yuheng Du \<[yuheng....@gmail.com](mailto:yuheng....@gmail.com) \<javascript:\>\> a  
> écrit :
> 
> BTW, the node stats info shows the jvm usage info as follows:  
> "jvm" : {  
> "timestamp" : 1409682903048,  
> "uptime\_in\_millis" : 488996,  
> "mem" : {  
> "heap\_used\_in\_bytes" : 48476472,  
> "heap\_used\_percent" : 9,  
> "heap\_committed\_in\_bytes" : 73412608,  
> "heap\_max\_in\_bytes" : 515571712,  
> "non\_heap\_used\_in\_bytes" : 57043440,  
> "non\_heap\_committed\_in\_bytes" : 92200960,  
> "pools" : {  
> "young" : {  
> "used\_in\_bytes" : 12126760,  
> "max\_in\_bytes" : 69795840,  
> "peak\_used\_in\_bytes" : 17367040,  
> "peak\_max\_in\_bytes" : 69795840  
> },  
> "survivor" : {  
> "used\_in\_bytes" : 915456,  
> "max\_in\_bytes" : 8716288,  
> "peak\_used\_in\_bytes" : 2097152,  
> "peak\_max\_in\_bytes" : 8716288  
> },  
> "old" : {  
> "used\_in\_bytes" : 35434256,  
> "max\_in\_bytes" : 437059584,  
> "peak\_used\_in\_bytes" : 38372208,  
> "peak\_max\_in\_bytes" : 437059584  
> }  
> }  
> },
> 
> On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:
> 
> > You have to start again the second node on the same machine to get the  
> > other shards back. Then when you cluster is yellow again, you can switch  
> > off the second node.
> > 
> > BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but  
> > this will do the same thing behind the scene). See  
> > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> > 
> > --  
> > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > 
> > Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > 
> > I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
> > Currently some of my shards are stated as "unassigned" in my kopf web  
> > interface. It happened after I started 3 nodes at the same time in my  
> > desktop yesterday.
> > 
> > There should be 3G of data in the cluster, but know I see only 1.92G. I  
> > think some data is stored in another node in my machine but I don't know  
> > how to fix it.
> > 
> > [https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
> > How can I fix this issue?
> > 
> > On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:
> > 
> > > Try 2g
> > > 
> > > --  
> > > David 😉  
> > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > 
> > > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > 
> > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > > 
> > > How can I deal with this?
> > > 
> > > Thanks
> > > 
> > > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> > > 
> > > > Ha! On your desktop? So that's not a production issue.
> > > > 
> > > > In that case, you should may be give more memory to all your  
> > > > elasticsearch nodes.  
> > > > What is your current HEAP size?
> > > > 
> > > > ```
> > > > -- 
> > > > 
> > > > ```
> > > > 
> > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > 
> > > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > écrit:
> > > > 
> > > > Each query is doing a search in \_all indexes in the cluster. Maybe  
> > > > that's also a hold up of the availability of ES server.
> > > > 
> > > > I tried to run 3 ES nodes on my desktop machine but it does not seems  
> > > > to help, it also crashed after some time.
> > > > 
> > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > > > 
> > > > > May be you could increase the number of replicas and nodes so your  
> > > > > cluster could handle more queries at a time.  
> > > > > The main cost is of course in term of machine and disk space.
> > > > > 
> > > > > Can't tell really more without more details about what you are doing  
> > > > > exactly. May be others have some ideas?
> > > > > 
> > > > > ```
> > > > > -- 
> > > > > 
> > > > > ```
> > > > > 
> > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > > 
> > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > > écrit:
> > > > > 
> > > > > Hi guys,
> > > > > 
> > > > > I have some streaming sensor data as input to ES. For each incoming  
> > > > > data message, I need to do a query on the historic data in ES according to  
> > > > > the 'timestamp' and 'messageId' in that message. I need to get the  
> > > > > aggregated query results in real-time.
> > > > > 
> > > > > My problem is each data message may sprout 28 queries at approximately  
> > > > > the same time and I get [No-nodes available] error from ES frequently.
> > > > > 
> > > > > Can anyone suggest what is the cause of this error and how can I fix  
> > > > > it? Or should I use some other system to handle this problem like apache  
> > > > > storm?
> > > > > 
> > > > > Thanks!
> > > > > 
> > > > > ## Yuheng
> > > > > 
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 7:25pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/13 "2014-09-02T19:25:05Z")

</div>

You don't have enough memory available on your laptop I guess...

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 21:05, Yuheng Du [yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com) a écrit :

Ok. you are right. I was able to set the maximum heap size and now it looks like:  
"jvm" : {  
"timestamp" : 1409684394790,  
"uptime\_in\_millis" : 128401,  
"mem" : {  
"heap\_used\_in\_bytes" : 520385872,  
"heap\_used\_percent" : 24,  
"heap\_committed\_in\_bytes" : 2138767360,  
"heap\_max\_in\_bytes" : 2138767360,  
"non\_heap\_used\_in\_bytes" : 46357584,  
"non\_heap\_committed\_in\_bytes" : 46727168,  
"pools" : {  
"young" : {  
"used\_in\_bytes" : 61936096,  
"max\_in\_bytes" : 69795840,  
"peak\_used\_in\_bytes" : 69795840,  
"peak\_max\_in\_bytes" : 69795840  
},  
"survivor" : {  
"used\_in\_bytes" : 8716288,  
"max\_in\_bytes" : 8716288,  
"peak\_used\_in\_bytes" : 8716288,  
"peak\_max\_in\_bytes" : 8716288  
},  
"old" : {  
"used\_in\_bytes" : 449733488,  
"max\_in\_bytes" : 2060255232,  
"peak\_used\_in\_bytes" : 449733488,  
"peak\_max\_in\_bytes" : 2060255232  
}  
}  
},

But when I start my two streaming data source which does the query, it simply get the ES instance killed. When I restart ES, I was told that:

OpenJDK 64-Bit Server VM warning: INFO: os::commit\_memory(0x000000077ad30000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)

# 

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 2060255232 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /home/yuheng/Downloads/elasticsearch-1.2.0/bin/hs\_err\_pid5116.log

Any thoughts?

> On Tuesday, September 2, 2014 2:48:17 PM UTC-4, David Pilato wrote:  
> I think your JVM settings are incorrect as we can see with heap\_max
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 2 sept. 2014 à 20:37, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> 
> BTW, the node stats info shows the jvm usage info as follows:  
> "jvm" : {  
> "timestamp" : 1409682903048,  
> "uptime\_in\_millis" : 488996,  
> "mem" : {  
> "heap\_used\_in\_bytes" : 48476472,  
> "heap\_used\_percent" : 9,  
> "heap\_committed\_in\_bytes" : 73412608,  
> "heap\_max\_in\_bytes" : 515571712,  
> "non\_heap\_used\_in\_bytes" : 57043440,  
> "non\_heap\_committed\_in\_bytes" : 92200960,  
> "pools" : {  
> "young" : {  
> "used\_in\_bytes" : 12126760,  
> "max\_in\_bytes" : 69795840,  
> "peak\_used\_in\_bytes" : 17367040,  
> "peak\_max\_in\_bytes" : 69795840  
> },  
> "survivor" : {  
> "used\_in\_bytes" : 915456,  
> "max\_in\_bytes" : 8716288,  
> "peak\_used\_in\_bytes" : 2097152,  
> "peak\_max\_in\_bytes" : 8716288  
> },  
> "old" : {  
> "used\_in\_bytes" : 35434256,  
> "max\_in\_bytes" : 437059584,  
> "peak\_used\_in\_bytes" : 38372208,  
> "peak\_max\_in\_bytes" : 437059584  
> }  
> }  
> },
> 
> > On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:  
> > You have to start again the second node on the same machine to get the other shards back. Then when you cluster is yellow again, you can switch off the second node.
> > 
> > BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this will do the same thing behind the scene). See [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> > 
> > --  
> > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > @dadoonet | @elasticsearchfr
> > 
> > > Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > 
> > > I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g. Currently some of my shards are stated as "unassigned" in my kopf web interface. It happened after I started 3 nodes at the same time in my desktop yesterday.
> > > 
> > > There should be 3G of data in the cluster, but know I see only 1.92G. I think some data is stored in another node in my machine but I don't know how to fix it.
> > > 
> > > How can I fix this issue?
> > > 
> > > > On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:  
> > > > Try 2g
> > > > 
> > > > --  
> > > > David 😉  
> > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > > 
> > > > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > > 
> > > > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > > > > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > > > > 
> > > > > How can I deal with this?
> > > > > 
> > > > > Thanks
> > > > > 
> > > > > > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:  
> > > > > > Ha! On your desktop? So that's not a production issue.
> > > > > > 
> > > > > > In that case, you should may be give more memory to all your elasticsearch nodes.  
> > > > > > What is your current HEAP size?
> > > > > > 
> > > > > > --  
> > > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > @dadoonet | @elasticsearchfr
> > > > > > 
> > > > > > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > > > > > 
> > > > > > > Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.
> > > > > > > 
> > > > > > > I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.
> > > > > > > 
> > > > > > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
> > > > > > > > May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
> > > > > > > > The main cost is of course in term of machine and disk space.
> > > > > > > > 
> > > > > > > > Can't tell really more without more details about what you are doing exactly. May be others have some ideas?
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > > > @dadoonet | @elasticsearchfr
> > > > > > > > 
> > > > > > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:
> > > > > > > > > 
> > > > > > > > > Hi guys,
> > > > > > > > > 
> > > > > > > > > I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.
> > > > > > > > > 
> > > > > > > > > My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.
> > > > > > > > > 
> > > > > > > > > Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?
> > > > > > > > > 
> > > > > > > > > Thanks!
> > > > > > > > > 
> > > > > > > > > ## Yuheng
> > > > > > > > > 
> > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com).  
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com).  
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com).  
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).  
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com).  
> > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/92054E4D-E995-49E4-A1CE-A4ED05365909%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/92054E4D-E995-49E4-A1CE-A4ED05365909%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 8:46pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/14 "2014-09-02T20:46:11Z")

</div>

Tried to increase the memory for the VM, now ES runs ok, but still gives me  
error:

[https://lh6.googleusercontent.com/-ut50-c987VU/VAYsS7OQIII/AAAAAAAAAOk/BqbGetezU5o/s1600/no\_nodes.jpg](https://lh6.googleusercontent.com/-ut50-c987VU/VAYsS7OQIII/AAAAAAAAAOk/BqbGetezU5o/s1600/no_nodes.jpg)  
Which is caused by the Elasticsearch's perl module's Try::Tiny module.

How to remove this kind of warning/error?

Thanks.

On Tuesday, September 2, 2014 3:25:30 PM UTC-4, David Pilato wrote:

> You don't have enough memory available on your laptop I guess...
> 
> --  
> David 😉  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> 
> Le 2 sept. 2014 à 21:05, Yuheng Du \<[yuheng....@gmail.com](mailto:yuheng....@gmail.com) \<javascript:\>\> a  
> écrit :
> 
> Ok. you are right. I was able to set the maximum heap size and now it  
> looks like:
> 
> "jvm" : {
> 
> "timestamp" : 1409684394790,
> 
> "uptime\_in\_millis" : 128401,
> 
> "mem" : {
> 
> "heap\_used\_in\_bytes" : 520385872,
> 
> "heap\_used\_percent" : 24,
> 
> "heap\_committed\_in\_bytes" : 2138767360,
> 
> "heap\_max\_in\_bytes" : 2138767360,
> 
> "non\_heap\_used\_in\_bytes" : 46357584,
> 
> "non\_heap\_committed\_in\_bytes" : 46727168,
> 
> "pools" : {
> 
> "young" : {
> 
> "used\_in\_bytes" : 61936096,
> 
> "max\_in\_bytes" : 69795840,
> 
> "peak\_used\_in\_bytes" : 69795840,
> 
> "peak\_max\_in\_bytes" : 69795840
> 
> },
> 
> "survivor" : {
> 
> "used\_in\_bytes" : 8716288,
> 
> "max\_in\_bytes" : 8716288,
> 
> "peak\_used\_in\_bytes" : 8716288,
> 
> "peak\_max\_in\_bytes" : 8716288
> 
> },
> 
> "old" : {
> 
> "used\_in\_bytes" : 449733488,
> 
> "max\_in\_bytes" : 2060255232,
> 
> "peak\_used\_in\_bytes" : 449733488,
> 
> "peak\_max\_in\_bytes" : 2060255232
> 
> }
> 
> }
> 
> },
> 
> But when I start my two streaming data source which does the query, it  
> simply get the ES instance killed. When I restart ES, I was told that:
> 
> OpenJDK 64-Bit Server VM warning: INFO:  
> os::commit\_memory(0x000000077ad30000, 2060255232, 0) failed; error='Cannot  
> allocate memory' (errno=12)
> 
> # 
> 
> # There is insufficient memory for the Java Runtime Environment to
> 
> continue.
> 
> # Native memory allocation (malloc) failed to allocate 2060255232 bytes
> 
> for committing reserved memory.
> 
> # An error report file with more information is saved as:
> 
> # /home/yuheng/Downloads/elasticsearch-1.2.0/bin/hs\_err\_pid5116.log
> 
> Any thoughts?
> 
> On Tuesday, September 2, 2014 2:48:17 PM UTC-4, David Pilato wrote:
> 
> > I think your JVM settings are incorrect as we can see with heap\_max
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 2 sept. 2014 à 20:37, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > 
> > BTW, the node stats info shows the jvm usage info as follows:  
> > "jvm" : {  
> > "timestamp" : 1409682903048,  
> > "uptime\_in\_millis" : 488996,  
> > "mem" : {  
> > "heap\_used\_in\_bytes" : 48476472,  
> > "heap\_used\_percent" : 9,  
> > "heap\_committed\_in\_bytes" : 73412608,  
> > "heap\_max\_in\_bytes" : 515571712,  
> > "non\_heap\_used\_in\_bytes" : 57043440,  
> > "non\_heap\_committed\_in\_bytes" : 92200960,  
> > "pools" : {  
> > "young" : {  
> > "used\_in\_bytes" : 12126760,  
> > "max\_in\_bytes" : 69795840,  
> > "peak\_used\_in\_bytes" : 17367040,  
> > "peak\_max\_in\_bytes" : 69795840  
> > },  
> > "survivor" : {  
> > "used\_in\_bytes" : 915456,  
> > "max\_in\_bytes" : 8716288,  
> > "peak\_used\_in\_bytes" : 2097152,  
> > "peak\_max\_in\_bytes" : 8716288  
> > },  
> > "old" : {  
> > "used\_in\_bytes" : 35434256,  
> > "max\_in\_bytes" : 437059584,  
> > "peak\_used\_in\_bytes" : 38372208,  
> > "peak\_max\_in\_bytes" : 437059584  
> > }  
> > }  
> > },
> > 
> > On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:
> > 
> > > You have to start again the second node on the same machine to get the  
> > > other shards back. Then when you cluster is yellow again, you can switch  
> > > off the second node.
> > > 
> > > BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but  
> > > this will do the same thing behind the scene). See  
> > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> > > 
> > > --  
> > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > 
> > > Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > écrit:
> > > 
> > > I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
> > > Currently some of my shards are stated as "unassigned" in my kopf web  
> > > interface. It happened after I started 3 nodes at the same time in my  
> > > desktop yesterday.
> > > 
> > > There should be 3G of data in the cluster, but know I see only 1.92G. I  
> > > think some data is stored in another node in my machine but I don't know  
> > > how to fix it.
> > > 
> > > [https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
> > > How can I fix this issue?
> > > 
> > > On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:
> > > 
> > > > Try 2g
> > > > 
> > > > --  
> > > > David 😉  
> > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > > 
> > > > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > > 
> > > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > > > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > > > 
> > > > How can I deal with this?
> > > > 
> > > > Thanks
> > > > 
> > > > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> > > > 
> > > > > Ha! On your desktop? So that's not a production issue.
> > > > > 
> > > > > In that case, you should may be give more memory to all your  
> > > > > elasticsearch nodes.  
> > > > > What is your current HEAP size?
> > > > > 
> > > > > ```
> > > > > -- 
> > > > > 
> > > > > ```
> > > > > 
> > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > > 
> > > > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > > écrit:
> > > > > 
> > > > > Each query is doing a search in \_all indexes in the cluster. Maybe  
> > > > > that's also a hold up of the availability of ES server.
> > > > > 
> > > > > I tried to run 3 ES nodes on my desktop machine but it does not seems  
> > > > > to help, it also crashed after some time.
> > > > > 
> > > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > > > > 
> > > > > > May be you could increase the number of replicas and nodes so your  
> > > > > > cluster could handle more queries at a time.  
> > > > > > The main cost is of course in term of machine and disk space.
> > > > > > 
> > > > > > Can't tell really more without more details about what you are  
> > > > > > doing exactly. May be others have some ideas?
> > > > > > 
> > > > > > ```
> > > > > > -- 
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > > > 
> > > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > > > écrit:
> > > > > > 
> > > > > > Hi guys,
> > > > > > 
> > > > > > I have some streaming sensor data as input to ES. For each incoming  
> > > > > > data message, I need to do a query on the historic data in ES according to  
> > > > > > the 'timestamp' and 'messageId' in that message. I need to get the  
> > > > > > aggregated query results in real-time.
> > > > > > 
> > > > > > My problem is each data message may sprout 28 queries at  
> > > > > > approximately the same time and I get [No-nodes available] error from ES  
> > > > > > frequently.
> > > > > > 
> > > > > > Can anyone suggest what is the cause of this error and how can I fix  
> > > > > > it? Or should I use some other system to handle this problem like apache  
> > > > > > storm?
> > > > > > 
> > > > > > Thanks!
> > > > > > 
> > > > > > ## Yuheng
> > > > > > 
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .  
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .  
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 2, 2014, 8:50pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/15 "2014-09-02T20:50:50Z")

</div>

I have no idea. I don't know the Perl Client.

Does this works?

curl localhost:9200

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 22:46:15, Yuheng Du ([yuheng.du.hust@gmail.com](mailto:yuheng.du.hust@gmail.com)) a écrit:

Tried to increase the memory for the VM, now ES runs ok, but still gives me error:

Which is caused by the Elasticsearch's perl module's Try::Tiny module.

How to remove this kind of warning/error?

Thanks.

On Tuesday, September 2, 2014 3:25:30 PM UTC-4, David Pilato wrote:  
You don't have enough memory available on your laptop I guess...

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 21:05, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :

Ok. you are right. I was able to set the maximum heap size and now it looks like:  
"jvm" : {  
"timestamp" : 1409684394790,  
"uptime\_in\_millis" : 128401,  
"mem" : {  
"heap\_used\_in\_bytes" : 520385872,  
"heap\_used\_percent" : 24,  
"heap\_committed\_in\_bytes" : 2138767360,  
"heap\_max\_in\_bytes" : 2138767360,  
"non\_heap\_used\_in\_bytes" : 46357584,  
"non\_heap\_committed\_in\_bytes" : 46727168,  
"pools" : {  
"young" : {  
"used\_in\_bytes" : 61936096,  
"max\_in\_bytes" : 69795840,  
"peak\_used\_in\_bytes" : 69795840,  
"peak\_max\_in\_bytes" : 69795840  
},  
"survivor" : {  
"used\_in\_bytes" : 8716288,  
"max\_in\_bytes" : 8716288,  
"peak\_used\_in\_bytes" : 8716288,  
"peak\_max\_in\_bytes" : 8716288  
},  
"old" : {  
"used\_in\_bytes" : 449733488,  
"max\_in\_bytes" : 2060255232,  
"peak\_used\_in\_bytes" : 449733488,  
"peak\_max\_in\_bytes" : 2060255232  
}  
}  
},

But when I start my two streaming data source which does the query, it simply get the ES instance killed. When I restart ES, I was told that:

OpenJDK 64-Bit Server VM warning: INFO: os::commit\_memory(0x000000077ad30000, 2060255232, 0) failed; error='Cannot allocate memory' (errno=12)

# 

# There is insufficient memory for the Java Runtime Environment to continue.

# Native memory allocation (malloc) failed to allocate 2060255232 bytes for committing reserved memory.

# An error report file with more information is saved as:

# /home/yuheng/Downloads/elasticsearch-1.2.0/bin/hs\_err\_pid5116.log

Any thoughts?

On Tuesday, September 2, 2014 2:48:17 PM UTC-4, David Pilato wrote:  
I think your JVM settings are incorrect as we can see with heap\_max

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 20:37, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :

BTW, the node stats info shows the jvm usage info as follows:  
"jvm" : {  
"timestamp" : 1409682903048,  
"uptime\_in\_millis" : 488996,  
"mem" : {  
"heap\_used\_in\_bytes" : 48476472,  
"heap\_used\_percent" : 9,  
"heap\_committed\_in\_bytes" : 73412608,  
"heap\_max\_in\_bytes" : 515571712,  
"non\_heap\_used\_in\_bytes" : 57043440,  
"non\_heap\_committed\_in\_bytes" : 92200960,  
"pools" : {  
"young" : {  
"used\_in\_bytes" : 12126760,  
"max\_in\_bytes" : 69795840,  
"peak\_used\_in\_bytes" : 17367040,  
"peak\_max\_in\_bytes" : 69795840  
},  
"survivor" : {  
"used\_in\_bytes" : 915456,  
"max\_in\_bytes" : 8716288,  
"peak\_used\_in\_bytes" : 2097152,  
"peak\_max\_in\_bytes" : 8716288  
},  
"old" : {  
"used\_in\_bytes" : 35434256,  
"max\_in\_bytes" : 437059584,  
"peak\_used\_in\_bytes" : 38372208,  
"peak\_max\_in\_bytes" : 437059584  
}  
}  
},

On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:  
You have to start again the second node on the same machine to get the other shards back. Then when you cluster is yellow again, you can switch off the second node.

BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but this will do the same thing behind the scene). See [http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#\_environment\_variables](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g. Currently some of my shards are stated as "unassigned" in my kopf web interface. It happened after I started 3 nodes at the same time in my desktop yesterday.

There should be 3G of data in the cluster, but know I see only 1.92G. I think some data is stored in another node in my machine but I don't know how to fix it.

How can I fix this issue?

On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:  
Try 2g

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :

I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,

How can I deal with this?

Thanks

On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:  
Ha! On your desktop? So that's not a production issue.

In that case, you should may be give more memory to all your elasticsearch nodes.  
What is your current HEAP size?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

Each query is doing a search in \_all indexes in the cluster. Maybe that's also a hold up of the availability of ES server.

I tried to run 3 ES nodes on my desktop machine but it does not seems to help, it also crashed after some time.

On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:  
May be you could increase the number of replicas and nodes so your cluster could handle more queries at a time.  
The main cost is of course in term of machine and disk space.

Can't tell really more without more details about what you are doing exactly. May be others have some ideas?

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a écrit:

Hi guys,

I have some streaming sensor data as input to ES. For each incoming data message, I need to do a query on the historic data in ES according to the 'timestamp' and 'messageId' in that message. I need to get the aggregated query results in real-time.

My problem is each data message may sprout 28 queries at approximately the same time and I get [No-nodes available] error from ES frequently.

Can anyone suggest what is the cause of this error and how can I fix it? Or should I use some other system to handle this problem like apache storm?

Thanks!

## Yuheng

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.54062daa.71f32454.d74c%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.54062daa.71f32454.d74c%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![yuheng](https://avatars.discourse-cdn.com/v4/letter/y/b5ac83/32.png) [@yuheng](https://discuss.elastic.co/u/yuheng)\
**Post date:** [September 2, 2014, 8:52pm UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/16 "2014-09-02T20:52:14Z")

</div>

Yes, This works fine. The query request will eventually get processed. It  
just throws this message.

Thanks.

On Tuesday, September 2, 2014 4:51:08 PM UTC-4, David Pilato wrote:

> I have no idea. I don't know the Perl Client.
> 
> Does this works?
> 
> curl localhost:9200
> 
> --  
> _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)_  
> @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> 
> Le 2 septembre 2014 à 22:46:15, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)  
> \<javascript:\>) a écrit:
> 
> Tried to increase the memory for the VM, now ES runs ok, but still gives  
> me error:
> 
> [https://lh6.googleusercontent.com/-ut50-c987VU/VAYsS7OQIII/AAAAAAAAAOk/BqbGetezU5o/s1600/no\_nodes.jpg](https://lh6.googleusercontent.com/-ut50-c987VU/VAYsS7OQIII/AAAAAAAAAOk/BqbGetezU5o/s1600/no_nodes.jpg)  
> Which is caused by the Elasticsearch's perl module's Try::Tiny module.
> 
> How to remove this kind of warning/error?
> 
> Thanks.
> 
> On Tuesday, September 2, 2014 3:25:30 PM UTC-4, David Pilato wrote:
> 
> > You don't have enough memory available on your laptop I guess...
> > 
> > --  
> > David 😉  
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > 
> > Le 2 sept. 2014 à 21:05, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > 
> > Ok. you are right. I was able to set the maximum heap size and now it  
> > looks like:
> > 
> > "jvm" : {
> > 
> > "timestamp" : 1409684394790,
> > 
> > "uptime\_in\_millis" : 128401,
> > 
> > "mem" : {
> > 
> > "heap\_used\_in\_bytes" : 520385872,
> > 
> > "heap\_used\_percent" : 24,
> > 
> > "heap\_committed\_in\_bytes" : 2138767360,
> > 
> > "heap\_max\_in\_bytes" : 2138767360,
> > 
> > "non\_heap\_used\_in\_bytes" : 46357584,
> > 
> > "non\_heap\_committed\_in\_bytes" : 46727168,
> > 
> > "pools" : {
> > 
> > "young" : {
> > 
> > "used\_in\_bytes" : 61936096,
> > 
> > "max\_in\_bytes" : 69795840,
> > 
> > "peak\_used\_in\_bytes" : 69795840,
> > 
> > "peak\_max\_in\_bytes" : 69795840
> > 
> > },
> > 
> > "survivor" : {
> > 
> > "used\_in\_bytes" : 8716288,
> > 
> > "max\_in\_bytes" : 8716288,
> > 
> > "peak\_used\_in\_bytes" : 8716288,
> > 
> > "peak\_max\_in\_bytes" : 8716288
> > 
> > },
> > 
> > "old" : {
> > 
> > "used\_in\_bytes" : 449733488,
> > 
> > "max\_in\_bytes" : 2060255232,
> > 
> > "peak\_used\_in\_bytes" : 449733488,
> > 
> > "peak\_max\_in\_bytes" : 2060255232
> > 
> > }
> > 
> > }
> > 
> > },
> > 
> > But when I start my two streaming data source which does the query, it  
> > simply get the ES instance killed. When I restart ES, I was told that:
> > 
> > OpenJDK 64-Bit Server VM warning: INFO:  
> > os::commit\_memory(0x000000077ad30000, 2060255232, 0) failed; error='Cannot  
> > allocate memory' (errno=12)
> > 
> > # 
> > 
> > # There is insufficient memory for the Java Runtime Environment to
> > 
> > continue.
> > 
> > # Native memory allocation (malloc) failed to allocate 2060255232 bytes
> > 
> > for committing reserved memory.
> > 
> > # An error report file with more information is saved as:
> > 
> > # /home/yuheng/Downloads/elasticsearch-1.2.0/bin/hs\_err\_pid5116.log
> > 
> > Any thoughts?
> > 
> > On Tuesday, September 2, 2014 2:48:17 PM UTC-4, David Pilato wrote:
> > 
> > > I think your JVM settings are incorrect as we can see with heap\_max
> > > 
> > > --  
> > > David 😉  
> > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > 
> > > Le 2 sept. 2014 à 20:37, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > 
> > > BTW, the node stats info shows the jvm usage info as follows:  
> > > "jvm" : {  
> > > "timestamp" : 1409682903048,  
> > > "uptime\_in\_millis" : 488996,  
> > > "mem" : {  
> > > "heap\_used\_in\_bytes" : 48476472,  
> > > "heap\_used\_percent" : 9,  
> > > "heap\_committed\_in\_bytes" : 73412608,  
> > > "heap\_max\_in\_bytes" : 515571712,  
> > > "non\_heap\_used\_in\_bytes" : 57043440,  
> > > "non\_heap\_committed\_in\_bytes" : 92200960,  
> > > "pools" : {  
> > > "young" : {  
> > > "used\_in\_bytes" : 12126760,  
> > > "max\_in\_bytes" : 69795840,  
> > > "peak\_used\_in\_bytes" : 17367040,  
> > > "peak\_max\_in\_bytes" : 69795840  
> > > },  
> > > "survivor" : {  
> > > "used\_in\_bytes" : 915456,  
> > > "max\_in\_bytes" : 8716288,  
> > > "peak\_used\_in\_bytes" : 2097152,  
> > > "peak\_max\_in\_bytes" : 8716288  
> > > },  
> > > "old" : {  
> > > "used\_in\_bytes" : 35434256,  
> > > "max\_in\_bytes" : 437059584,  
> > > "peak\_used\_in\_bytes" : 38372208,  
> > > "peak\_max\_in\_bytes" : 437059584  
> > > }  
> > > }  
> > > },
> > > 
> > > On Tuesday, September 2, 2014 10:26:29 AM UTC-4, David Pilato wrote:
> > > 
> > > > You have to start again the second node on the same machine to get  
> > > > the other shards back. Then when you cluster is yellow again, you can  
> > > > switch off the second node.
> > > > 
> > > > BTW, you should set ES\_HEAP\_SIZE instead of dealing with Xmx Xms (but  
> > > > this will do the same thing behind the scene). See  
> > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html#_environment_variables)
> > > > 
> > > > ```
> > > > -- 
> > > > 
> > > > ```
> > > > 
> > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > 
> > > > Le 2 septembre 2014 à 16:14:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > écrit:
> > > > 
> > > > I tired 2g, It only worked when I use ./elasticsearch -Xmx2g -Xms2g.  
> > > > Currently some of my shards are stated as "unassigned" in my kopf web  
> > > > interface. It happened after I started 3 nodes at the same time in my  
> > > > desktop yesterday.
> > > > 
> > > > There should be 3G of data in the cluster, but know I see only 1.92G. I  
> > > > think some data is stored in another node in my machine but I don't know  
> > > > how to fix it.
> > > > 
> > > > [https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg](https://lh3.googleusercontent.com/-RZb2tkusmlc/VAXQqoCriOI/AAAAAAAAAOE/omiptkABSuU/s1600/kopf.jpg)  
> > > > How can I fix this issue?
> > > > 
> > > > On Tuesday, September 2, 2014 10:04:40 AM UTC-4, David Pilato wrote:
> > > > 
> > > > > Try 2g
> > > > > 
> > > > > --  
> > > > > David 😉  
> > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > > > 
> > > > > Le 2 sept. 2014 à 15:49, Yuheng Du [yuheng....@gmail.com](mailto:yuheng....@gmail.com) a écrit :
> > > > > 
> > > > > I tried to run ./elasticsearch --node.name=esmaster -Xmx=2G -Xms=2G.  
> > > > > But it gives me the error msg: Invalid maximum heap size: -Xmx=2G,
> > > > > 
> > > > > How can I deal with this?
> > > > > 
> > > > > Thanks
> > > > > 
> > > > > On Tuesday, September 2, 2014 3:46:26 AM UTC-4, David Pilato wrote:
> > > > > 
> > > > > > Ha! On your desktop? So that's not a production issue.
> > > > > > 
> > > > > > In that case, you should may be give more memory to all your  
> > > > > > elasticsearch nodes.  
> > > > > > What is your current HEAP size?
> > > > > > 
> > > > > > ```
> > > > > > -- 
> > > > > > 
> > > > > > ```
> > > > > > 
> > > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > > > 
> > > > > > Le 2 septembre 2014 à 09:17:31, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > > > écrit:
> > > > > > 
> > > > > > Each query is doing a search in \_all indexes in the cluster. Maybe  
> > > > > > that's also a hold up of the availability of ES server.
> > > > > > 
> > > > > > I tried to run 3 ES nodes on my desktop machine but it does not seems  
> > > > > > to help, it also crashed after some time.
> > > > > > 
> > > > > > On Tuesday, September 2, 2014 3:10:13 AM UTC-4, David Pilato wrote:
> > > > > > 
> > > > > > > May be you could increase the number of replicas and nodes so your  
> > > > > > > cluster could handle more queries at a time.  
> > > > > > > The main cost is of course in term of machine and disk space.
> > > > > > > 
> > > > > > > Can't tell really more without more details about what you are  
> > > > > > > doing exactly. May be others have some ideas?
> > > > > > > 
> > > > > > > ```
> > > > > > > -- 
> > > > > > > 
> > > > > > > ```
> > > > > > > 
> > > > > > > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com)  
> > > > > > > [http://Elasticsearch.com](http://Elasticsearch.com)_  
> > > > > > > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr  
> > > > > > > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr)
> > > > > > > 
> > > > > > > Le 2 septembre 2014 à 08:53:13, Yuheng Du ([yuheng....@gmail.com](mailto:yuheng....@gmail.com)) a  
> > > > > > > écrit:
> > > > > > > 
> > > > > > > Hi guys,
> > > > > > > 
> > > > > > > I have some streaming sensor data as input to ES. For each incoming  
> > > > > > > data message, I need to do a query on the historic data in ES according to  
> > > > > > > the 'timestamp' and 'messageId' in that message. I need to get the  
> > > > > > > aggregated query results in real-time.
> > > > > > > 
> > > > > > > My problem is each data message may sprout 28 queries at  
> > > > > > > approximately the same time and I get [No-nodes available] error from ES  
> > > > > > > frequently.
> > > > > > > 
> > > > > > > Can anyone suggest what is the cause of this error and how can I fix  
> > > > > > > it? Or should I use some other system to handle this problem like apache  
> > > > > > > storm?
> > > > > > > 
> > > > > > > Thanks!
> > > > > > > 
> > > > > > > ## Yuheng
> > > > > > > 
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com)  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6dace925-d2d8-443f-bc9d-1e333ef96890%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > .  
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > > 
> > > > > > > --  
> > > > > > > You received this message because you are subscribed to the Google  
> > > > > > > Groups "elasticsearch" group.  
> > > > > > > To unsubscribe from this group and stop receiving emails from it,  
> > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > > To view this discussion on the web visit  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com)  
> > > > > > > [https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/476fd9e6-71ee-4e14-9bd0-dd3cd1cc42ca%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > > .  
> > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > > 
> > > > > > --  
> > > > > > You received this message because you are subscribed to the Google  
> > > > > > Groups "elasticsearch" group.  
> > > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > > To view this discussion on the web visit  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com)  
> > > > > > [https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5fea40c8-7645-48e5-8d2a-02e18f5ef7cf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > > .  
> > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > > 
> > > > > --  
> > > > > You received this message because you are subscribed to the Google  
> > > > > Groups "elasticsearch" group.  
> > > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > > To view this discussion on the web visit  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com)  
> > > > > [https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4b7432b3-3956-42c3-a258-ff596148157c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > > .  
> > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > > > 
> > > > --  
> > > > You received this message because you are subscribed to the Google  
> > > > Groups "elasticsearch" group.  
> > > > To unsubscribe from this group and stop receiving emails from it, send  
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > > > To view this discussion on the web visit  
> > > > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com)  
> > > > [https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/82b1caf1-eadd-44c8-89e8-a0393cf45abf%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > > .  
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/415adbee-424a-400e-82e4-382275de6d3b%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4891b5a1-5c3a-4d9e-85b7-5442707b9438%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/56607e18-99f3-4591-b370-2b42956d9b2b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/56607e18-99f3-4591-b370-2b42956d9b2b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:04am UTC](https://discuss.elastic.co/t/need-help-suggestion-for-the-massive-queries-user-case/19567/17 "2017-07-06T01:04:52Z")

</div>