Searching the big index - java.lang.OutOfMemoryError: Java heap space


(Pavel P) #1

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform such
queries? Which configuration should I have and how to tweak the current?

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/726b0315-a2cf-4537-a24f-f18bffaf0eda%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

What is the type of query you execute, how does it look like? Can you give
an example?

Does it work with standard settings, without manipulating cache/filter size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works
well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform such
queries? Which configuration should I have and how to tweak the current?

Regards,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Pavel P) #3

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started to
fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query, it
tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante joergprante@gmail.com wrote:

What is the type of query you execute, how does it look like? Can you give
an example?

Does it work with standard settings, without manipulating cache/filter
size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works
well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform such
queries? Which configuration should I have and how to tweak the current?

Regards,

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027


facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #4

You should use a filter for timestamp, not a query. If you use timestamp
range plus boolean and, this is expensive.

Jörg

On Fri, Aug 8, 2014 at 10:12 AM, Pavel P pavel@kredito.de wrote:

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started to
fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query, it
tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante joergprante@gmail.com
wrote:

What is the type of query you execute, how does it look like? Can you
give an example?

Does it work with standard settings, without manipulating cache/filter
size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works
well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform
such queries? Which configuration should I have and how to tweak the
current?

Regards,

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Pavel P) #5

2Jörg

Unfortunately, I had problems with it.

Because the timestamp is saved with the timezone (as I understand).

And the time fileter is based on my current timezone, so the dates are not
correlating.

[image: Inline image 2]

Therefore I decided to put those values in the query directly.

But I got your point, would have it in mind.

Would it helps, if I add some nodes to the cluster? Because it look like
currently, any customer with heavy query could put our cluster down. Or how
could we make it more error prune?

Thanks!

On Fri, Aug 8, 2014 at 11:18 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You should use a filter for timestamp, not a query. If you use timestamp
range plus boolean and, this is expensive.

Jörg

On Fri, Aug 8, 2014 at 10:12 AM, Pavel P pavel@kredito.de wrote:

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started to
fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query, it
tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante joergprante@gmail.com
wrote:

What is the type of query you execute, how does it look like? Can you
give an example?

Does it work with standard settings, without manipulating cache/filter
size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works
well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform
such queries? Which configuration should I have and how to tweak the
current?

Regards,

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027


facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqOo1LVLcKjd%2BaKjapCGRVsM9-OUPnDnZCV2QsEg%3DQveuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Pavel P) #6

2Jörg

Do you also think that I should use the default values for:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

and so on?

On Fri, Aug 8, 2014 at 11:43 AM, Pavel P pavel@kredito.de wrote:

2Jörg

Unfortunately, I had problems with it.

Because the timestamp is saved with the timezone (as I understand).

And the time fileter is based on my current timezone, so the dates are not
correlating.

[image: Inline image 2]

Therefore I decided to put those values in the query directly.

But I got your point, would have it in mind.

Would it helps, if I add some nodes to the cluster? Because it look like
currently, any customer with heavy query could put our cluster down. Or how
could we make it more error prune?

Thanks!

On Fri, Aug 8, 2014 at 11:18 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You should use a filter for timestamp, not a query. If you use timestamp
range plus boolean and, this is expensive.

Jörg

On Fri, Aug 8, 2014 at 10:12 AM, Pavel P pavel@kredito.de wrote:

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started to
fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query,
it tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante joergprante@gmail.com
wrote:

What is the type of query you execute, how does it look like? Can you
give an example?

Does it work with standard settings, without manipulating cache/filter
size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb
ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing works
well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform
such queries? Which configuration should I have and how to tweak the
current?

Regards,

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027


facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFVUaqPNGynzBhxmxOw5iXeNTfZb32_0gRgbnTPjSh2ZyUL5PA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Pavel P) #7

In my case it seems that I've solved after this configuration:

/etc/default/elasticsearch

Run Elasticsearch as this user ID and group ID

#ES_USER=elasticsearch
#ES_GROUP=elasticsearch

Heap Size (defaults to 256m min, 1g max)

ES_HEAP_SIZE=15g # ES developers advice to set this as 50% of available memory

Heap new generation

#ES_HEAP_NEWSIZE=

max direct memory

#ES_DIRECT_SIZE=

Maximum number of open files, defaults to 65535.

#MAX_OPEN_FILES=65535

Maximum locked memory size. Set to "unlimited" if you use the

bootstrap.mlockall option in elasticsearch.yml. You must also set

ES_HEAP_SIZE.

MAX_LOCKED_MEMORY=unlimited

Before that the ES_HEAP_SIZE was the default one - 1g

btw, if someone knows how exactly we should point the measures in configs, please, drop a line here

it seems that in /etc/default/elasticsearch we use "g" as Gb, while in the /etc/elasticsearch/elasticsearch.yml we use "Gb".
It's not clear where should we use G, g or Gb, or we can use any of them in any place.

Regards,

On Friday, August 8, 2014 11:54:01 AM UTC+3, Pavel P wrote:

2Jörg

Do you also think that I should use the default values for:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

and so on?

On Fri, Aug 8, 2014 at 11:43 AM, Pavel P pavel@kredito.de wrote:

2Jörg

Unfortunately, I had problems with it.

Because the timestamp is saved with the timezone (as I understand).

And the time fileter is based on my current timezone, so the dates are
not correlating.

[image: Inline image 2]

Therefore I decided to put those values in the query directly.

But I got your point, would have it in mind.

Would it helps, if I add some nodes to the cluster? Because it look like
currently, any customer with heavy query could put our cluster down. Or how
could we make it more error prune?

Thanks!

On Fri, Aug 8, 2014 at 11:18 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

You should use a filter for timestamp, not a query. If you use timestamp
range plus boolean and, this is expensive.

Jörg

On Fri, Aug 8, 2014 at 10:12 AM, Pavel P pavel@kredito.de wrote:

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started
to fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query,
it tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante joergprante@gmail.com
wrote:

What is the type of query you execute, how does it look like? Can you
give an example?

Does it work with standard settings, without manipulating cache/filter
size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb
ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing
works well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform
such queries? Which configuration should I have and how to tweak the
current?

Regards,

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pavel@kredito.de
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information. If
you are not the intended recipient or if you have received this e-mail by
error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7bdf7b08-18bf-406b-8e69-9e8c15f8166e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(James-4) #8

This was helpful thanks Pavel.

On Friday, August 8, 2014 5:16:46 AM UTC-6, Pavel P wrote:

In my case it seems that I've solved after this configuration:

/etc/default/elasticsearch

Run Elasticsearch as this user ID and group ID

#ES_USER=elasticsearch
#ES_GROUP=elasticsearch

Heap Size (defaults to 256m min, 1g max)

ES_HEAP_SIZE=15g # ES developers advice to set this as 50% of available memory

Heap new generation

#ES_HEAP_NEWSIZE=

max direct memory

#ES_DIRECT_SIZE=

Maximum number of open files, defaults to 65535.

#MAX_OPEN_FILES=65535

Maximum locked memory size. Set to "unlimited" if you use the

bootstrap.mlockall option in elasticsearch.yml. You must also set

ES_HEAP_SIZE.

MAX_LOCKED_MEMORY=unlimited

Before that the ES_HEAP_SIZE was the default one - 1g

btw, if someone knows how exactly we should point the measures in configs, please, drop a line here

it seems that in /etc/default/elasticsearch we use "g" as Gb, while in the /etc/elasticsearch/elasticsearch.yml we use "Gb".
It's not clear where should we use G, g or Gb, or we can use any of them in any place.

Regards,

On Friday, August 8, 2014 11:54:01 AM UTC+3, Pavel P wrote:

2Jörg

Do you also think that I should use the default values for:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

and so on?

On Fri, Aug 8, 2014 at 11:43 AM, Pavel P <pa...@kredito.de <javascript:>>
wrote:

2Jörg

Unfortunately, I had problems with it.

Because the timestamp is saved with the timezone (as I understand).

And the time fileter is based on my current timezone, so the dates are
not correlating.

[image: Inline image 2]

Therefore I decided to put those values in the query directly.

But I got your point, would have it in mind.

Would it helps, if I add some nodes to the cluster? Because it look like
currently, any customer with heavy query could put our cluster down. Or how
could we make it more error prune?

Thanks!

On Fri, Aug 8, 2014 at 11:18 AM, joerg...@gmail.com <javascript:> <
joerg...@gmail.com <javascript:>> wrote:

You should use a filter for timestamp, not a query. If you use
timestamp range plus boolean and, this is expensive.

Jörg

On Fri, Aug 8, 2014 at 10:12 AM, Pavel P <pa...@kredito.de
<javascript:>> wrote:

Hi,

The query is the next:
[image: Inline image 1]

No, as the general size of the open indexes grew to 290Gb, it started
to fail for the queries. So I assume it doesn't work for the default values.

It even fails when I try to filter the index by _type value only, like
clicking here:
[image: Inline image 2]

I can spot it only in the node logs. After I try to perform the query,
it tries to perform it and the spinners keep running.
The in the logs I see the "heap" exception, and the cluster doesn't
response at all (for the flush from the logstash, for example).

Any thoughts?

On Fri, Aug 8, 2014 at 11:06 AM, Jörg Prante <joerg...@gmail.com
<javascript:>> wrote:

What is the type of query you execute, how does it look like? Can you
give an example?

Does it work with standard settings, without manipulating
cache/filter size?

Jörg

On Thursday, August 7, 2014 9:51:38 PM UTC+2, Pavel P wrote:

Hi,

Currently I have the cluster from 3 nodes, each has 8 CPUs and 30Gb
ram.

https://lh4.googleusercontent.com/-r-mHzuBN0e8/U-PYE0sNIrI/AAAAAAAAAIc/mliNQ7MDVmQ/s1600/Screen+Shot+2014-08-07+at+10.48.11+PM.png

Each day my logstash produces the 40Gb index there. The indexing
works well.

However, when I try to search it - the node fails and dies with the
exception:

Caused by: org.elasticsearch.ElasticsearchException:
java.lang.OutOfMemoryError: Java heap space

In my config file I've set the next values already:

indices.fielddata.cache.size: "10Gb"
indices.cache.filter.size: "10Gb"

Also I've executed the next query:

curl -XPUT http://host.com:9200/_cluster/settings -d '
{
"persistent" : {
"indices.fielddata.breaker.limit" : 40%
}
}'

But the query fails still, and I need to restart the node manually.

Could someone explain, how should I configure the cluster to perform
such queries? Which configuration should I have and how to tweak the
current?

Regards,

--
You received this message because you are subscribed to a topic in
the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eb1e5bce-e0a6-4f2d-81a6-56f50b570ee9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pa...@kredito.de <javascript:>
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected
information. If you are not the intended recipient or if you have received
this e-mail by error please notify the sender immediately and destroy this
e-mail. Any unauthorized review, copying, disclosure or distribution of the
material in this e-mail is strictly forbidden. The contents of this e-mail
is legally binding only if it is confirmed by letter or fax. The sending of
e-mails to us does not have any period-protecting effect. Thank you for
your cooperation.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAFVUaqNwYC23Bv0BYXap67kn7WoQkH50Uz70%3DyqoGGLzugJiDQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jbHg99gqRf0/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGBFCCw04HTbnH0vysknuxucwyi8xZTjQxoVm56FJeTBw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pa...@kredito.de <javascript:>
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--

Pavel Polyakov

Software Engineer - PHP team

E-mail: pa...@kredito.de <javascript:>
Skype: pavel.polyakov.x1

https://www.facebook.com/kreditech
Kreditech Holding SSL GmbH
Am Sandtorkai 50, 20457 Hamburg, Germany
Office phone: +49 (0)40 - 605905-60
Authorized representatives: Sebastian Diemer, Alexander Graubner-Müller
Company registration: Hamburg HRB122027

www.kreditech.com
facebook.com/kreditech https://www.facebook.com/kreditech

https://www.facebook.com/kreditech

This e-mail contains confidential and/or legally protected information.
If you are not the intended recipient or if you have received this e-mail
by error please notify the sender immediately and destroy this e-mail. Any
unauthorized review, copying, disclosure or distribution of the material in
this e-mail is strictly forbidden. The contents of this e-mail is legally
binding only if it is confirmed by letter or fax. The sending of e-mails to
us does not have any period-protecting effect. Thank you for your
cooperation.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/acd3c9c1-58bd-4827-b57c-b935660bc6b1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #9