Loading field caused out of memory failure

Elastic Stack Elasticsearch
1

Hi,

I've a strange failure on my ES cluster. I've 3 nodes with a daily index(4
actually) to store server's logs. When I execute this query:

{"query" : {"match_all" : {}},"facets" : {"Logs" : {"terms" : {"field" :
"tag","size" : 10, "order" : "term"}}}}

I've a succesfull answer. But I would modify the query and replace "tag" by
this field name "req_url". When I perform this query, I've a timeout error
and if I check my ES logs in a node, I can see that:

[2013-04-11 12:17:21,330][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
[2013-04-11 12:17:21,349][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space

And on another node I've these logs:

[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][1], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.08][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,399][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][0], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,401][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,402][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][3], node[f6aKvGEVR7CyJ7G-o3WsGA],
[P], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected

[2013-04-11 12:22:04,527][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,528][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,843][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,844][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,846][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,173][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,174][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,376][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]

The status of my ES cluster became Red. I'm beginner in the ES world, it's
possible that I perform a bad request but I find the impact very huge!

Thanks in advance

Matthieu

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Yes, the impact of performing a search request on a new facet field if
there is not enough memory is big. You have ran out of Java heap space. You
need to increase it and you can do this via the ES_HEAP_SIZE environment
variable. This env variable is set to 1GB by default. I'd double it and see
if it is sufficient. In any case don't set it higher than ~half of your
available RAM.

I also recommend setting up warming queries. Configure warming queries that
facet / sort on all the field your users would use. This way you find out
that you don't have enough memory at startup. Also when using warming
queries your users won't notice when field values are loaded in memory,
since this would happen as part of the refresh and not as part of the
search.

On 11 April 2013 12:46, Matthieu mboret86@gmail.com wrote:

Hi,

I've a strange failure on my ES cluster. I've 3 nodes with a daily index(4
actually) to store server's logs. When I execute this query:

{"query" : {"match_all" : {}},"facets" : {"Logs" : {"terms" : {"field" :
"tag","size" : 10, "order" : "term"}}}}

I've a succesfull answer. But I would modify the query and replace "tag"
by this field name "req_url". When I perform this query, I've a timeout
error and if I check my ES logs in a node, I can see that:

[2013-04-11 12:17:21,330][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
[2013-04-11 12:17:21,349][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space

And on another node I've these logs:

[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][1], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.08][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,399][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][0], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,401][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,402][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][3], node[f6aKvGEVR7CyJ7G-o3WsGA],
[P], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected

[2013-04-11 12:22:04,527][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,528][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,843][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,844][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,846][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,173][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,174][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,376][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]

The status of my ES cluster became Red. I'm beginner in the ES world, it's
possible that I perform a bad request but I find the impact very huge!

Thanks in advance

Matthieu

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Martijin,

Thanks a lot for your answer. You're right Java doesn't start with the
value ES_MIN_MEM and ES_MAX_MEM which have been setup in the eleasticsearch
init.d. I've correct this error and this avoid my OOM failure.
But can you give me more informations about your warning query? It's the
same that warm-up query for solr?

Thanks

Matthieu

On Thursday, April 11, 2013 3:07:03 PM UTC+2, Martijn v Groningen wrote:

Yes, the impact of performing a search request on a new facet field if
there is not enough memory is big. You have ran out of Java heap space. You
need to increase it and you can do this via the ES_HEAP_SIZE environment
variable. This env variable is set to 1GB by default. I'd double it and see
if it is sufficient. In any case don't set it higher than ~half of your
available RAM.

I also recommend setting up warming queries. Configure warming queries
that facet / sort on all the field your users would use. This way you find
out that you don't have enough memory at startup. Also when using warming
queries your users won't notice when field values are loaded in memory,
since this would happen as part of the refresh and not as part of the
search.

On 11 April 2013 12:46, Matthieu <mbor...@gmail.com <javascript:>> wrote:

Hi,

I've a strange failure on my ES cluster. I've 3 nodes with a daily
index(4 actually) to store server's logs. When I execute this query:

{"query" : {"match_all" : {}},"facets" : {"Logs" : {"terms" : {"field" :
"tag","size" : 10, "order" : "term"}}}}

I've a succesfull answer. But I would modify the query and replace "tag"
by this field name "req_url". When I perform this query, I've a timeout
error and if I check my ES logs in a node, I can see that:

[2013-04-11 12:17:21,330][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
[2013-04-11 12:17:21,349][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space

And on another node I've these logs:

[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][1], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.08][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,399][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][0], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,401][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected
[2013-04-11 12:14:43,402][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][3], node[f6aKvGEVR7CyJ7G-o3WsGA],
[P], s[STARTED]: Failed to execute
[org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@6eee124]
org.elasticsearch.transport.NodeDisconnectedException:
[mor-elasticsearch-02][inet[/192.168.245.100:9301]][indices/stats/s]
disconnected

[2013-04-11 12:22:04,527][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,528][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,843][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,844][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:04,846][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,173][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,174][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]
[2013-04-11 12:22:05,376][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/192.168.245.99:9300]]
marked shard as started, but shard have not been created, mark shard as
failed]

The status of my ES cluster became Red. I'm beginner in the ES world,
it's possible that I perform a bad request but I find the impact very huge!

Thanks in advance

Matthieu

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

The warming api is similar to Solr's warmup query.
The warming api's documentation page describes how your can use it:

On 11 April 2013 16:34, Matthieu mboret86@gmail.com wrote:

Hi Martijin,

Thanks a lot for your answer. You're right Java doesn't start with the
value ES_MIN_MEM and ES_MAX_MEM which have been setup in the eleasticsearch
init.d. I've correct this error and this avoid my OOM failure.
But can you give me more informations about your warning query? It's the
same that warm-up query for solr?

Thanks

Matthieu

On Thursday, April 11, 2013 3:07:03 PM UTC+2, Martijn v Groningen wrote:

Yes, the impact of performing a search request on a new facet field if
there is not enough memory is big. You have ran out of Java heap space. You
need to increase it and you can do this via the ES_HEAP_SIZE environment
variable. This env variable is set to 1GB by default. I'd double it and see
if it is sufficient. In any case don't set it higher than ~half of your
available RAM.

I also recommend setting up warming queries. Configure warming queries
that facet / sort on all the field your users would use. This way you find
out that you don't have enough memory at startup. Also when using warming
queries your users won't notice when field values are loaded in memory,
since this would happen as part of the refresh and not as part of the
search.

On 11 April 2013 12:46, Matthieu mbor...@gmail.com wrote:

Hi,

I've a strange failure on my ES cluster. I've 3 nodes with a daily
index(4 actually) to store server's logs. When I execute this query:

{"query" : {"match_all" : {}},"facets" : {"Logs" : {"terms" : {"field" :
"tag","size" : 10, "order" : "term"}}}}

I've a succesfull answer. But I would modify the query and replace "tag"
by this field name "req_url". When I perform this query, I've a timeout
error and if I check my ES logs in a node, I can see that:

[2013-04-11 12:17:21,330][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
[2013-04-11 12:17:21,349][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space

And on another node I've these logs:

[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][1], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.08][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,399][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][0], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,401][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,402][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][3], node[f6aKvGEVR7CyJ7G-o3WsGA],
[P], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected

[2013-04-11 12:22:04,527][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,528][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,843][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,844][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,846][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,173][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,174][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,376][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]

The status of my ES cluster became Red. I'm beginner in the ES world,
it's possible that I perform a bad request but I find the impact very huge!

Thanks in advance

Matthieu

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Met vriendelijke groet,

Martijn van Groningen

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Ok, Thanks for all Martijn.

Matthieu

On Thursday, April 11, 2013 4:52:55 PM UTC+2, Martijn v Groningen wrote:

The warming api is similar to Solr's warmup query.
The warming api's documentation page describes how your can use it:
Elasticsearch Platform — Find real-time answers at scale | Elastic

On 11 April 2013 16:34, Matthieu <mbor...@gmail.com <javascript:>> wrote:

Hi Martijin,

Thanks a lot for your answer. You're right Java doesn't start with the
value ES_MIN_MEM and ES_MAX_MEM which have been setup in the eleasticsearch
init.d. I've correct this error and this avoid my OOM failure.
But can you give me more informations about your warning query? It's the
same that warm-up query for solr?

Thanks

Matthieu

On Thursday, April 11, 2013 3:07:03 PM UTC+2, Martijn v Groningen wrote:

Yes, the impact of performing a search request on a new facet field if
there is not enough memory is big. You have ran out of Java heap space. You
need to increase it and you can do this via the ES_HEAP_SIZE environment
variable. This env variable is set to 1GB by default. I'd double it and see
if it is sufficient. In any case don't set it higher than ~half of your
available RAM.

I also recommend setting up warming queries. Configure warming queries
that facet / sort on all the field your users would use. This way you find
out that you don't have enough memory at startup. Also when using warming
queries your users won't notice when field values are loaded in memory,
since this would happen as part of the refresh and not as part of the
search.

On 11 April 2013 12:46, Matthieu mbor...@gmail.com wrote:

Hi,

I've a strange failure on my ES cluster. I've 3 nodes with a daily
index(4 actually) to store server's logs. When I execute this query:

{"query" : {"match_all" : {}},"facets" : {"Logs" : {"terms" : {"field"
: "tag","size" : 10, "order" : "term"}}}}

I've a succesfull answer. But I would modify the query and replace
"tag" by this field name "req_url". When I perform this query, I've a
timeout error and if I check my ES logs in a node, I can see that:

[2013-04-11 12:17:21,330][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space
[2013-04-11 12:17:21,349][WARN ][index.cache.field.data.soft]
[mor-elasticsearch-02] [logs-2013.04.10] loading field [req_url] caused out
of memory failure
java.lang.OutOfMemoryError: Java heap space

And on another node I've these logs:

[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][1], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,398][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.08][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,399][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][0], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,401][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][2], node[f6aKvGEVR7CyJ7G-o3WsGA],
[R], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected
[2013-04-11 12:14:43,402][DEBUG][action.admin.indices.stats]
[mor-elasticsearch-03] [logs-2013.04.10][3], node[f6aKvGEVR7CyJ7G-o3WsGA],
[P], s[STARTED]: Failed to execute [org.elasticsearch.action.
admin.indices.stats.**IndicesStatsRequest@6eee124]
org.elasticsearch.transport.**NodeDisconnectedException:
[mor-elasticsearch-02][inet[/**192.168.245.100:9301]][**indices/stats/s]
disconnected

[2013-04-11 12:22:04,527][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,528][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,843][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,844][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][1],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:04,846][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,173][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,174][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.08][3],
node[nYV56WwyQPaAwZW7bORM4A], [P], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]
[2013-04-11 12:22:05,376][WARN ][cluster.action.shard ]
[mor-elasticsearch-01] received shard failed for [logs-2013.04.09][4],
node[nYV56WwyQPaAwZW7bORM4A], [R], s[STARTED], reason [master
[mor-elasticsearch-01][DdmTythfSo25UkWPcWgxAg][inet[/
192.168.245.99:9300 http://192.168.245.99:9300]] marked shard as
started, but shard have not been created, mark shard as failed]

The status of my ES cluster became Red. I'm beginner in the ES world,
it's possible that I perform a bad request but I find the impact very huge!

Thanks in advance

Matthieu

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Met vriendelijke groet,

Martijn van Groningen

--
Met vriendelijke groet,

Martijn van Groningen

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.