Org.elasticsearch.common.breaker.CircuitBreakingException

Elastic Stack Elasticsearch
1

Hello,

we are running a ES cluster (v1.7) in a production environment. We are indexing network device syslogs.
The syslogs are parsed and indexed and it is possible to search them by date.

the mapping for the date part is:

                "date": {
                    "type": "date",
                    "index": "analyzed",
                    "store": true,
                    "format": "date_time_no_millis||yyyy-MM-dd HH:mm:ss||yyyy-MM-dd'T'HH:mm:ss"
                }

Today we are facing and issue where it is not possible to sort the search result by date.
For example, the query:

curl -XPOST 'http://10.250.131.216:9200/ubilogs-15.2.2/logs/_search?' -d ' {
"fields": [
"_id",
"_timestamp",
"_source"
],
"query": {"bool": {
"must": [
{"query_string": {
"default_field": "rawlog",
"query": "_id:AVMMO9EI4dV_htXhBe5Y",
"default_operator": "AND"
}}
],
"should": [],
"must_not": []
}},
"from": 0,
"size": 120
}
'

is returning one result, which is expected since this is a search by _id:
{
"took": 8,
"timed_out": false,
"_shards": {
"total": 20,
"successful": 20,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 18.160692,
"hits": [{
"_index": "ubilogs-15.2.2",
"_type": "logs",
"_id": "AVMMO9EI4dV_htXhBe5Y",
"_score": 18.160692,
"_source": {
"_ttl": "104w",
"rawlog": "%VNOC-1-PUSHCONFIG: ! delete "Web-Test_10_0_111_10" @Command fail@ entry 'Web-Test_10_0_111_10' not found Command fail. Entry not found. NCB565 (server-pool) ",
"severity": "1",
"customer_ref": "N151015039_CUS-JP-00570401",
"customer_id": "212",
"man_id": "17",
"mod_id": "1130",
"date": "2016-02-23 12:45:59",
"device_id": "NCB565",
"hostname": "NCB565",
"type": "VNOC",
"subtype": "PUSHCONFIG"
},
"fields": {
"_timestamp": 1456199160072
}
}]
}
}

but when we add a sort by date : http://10.250.131.216:9200/ubilogs-15.2.2/logs/_search?sort=date
we get the error below in the log file:

org.elasticsearch.transport.RemoteTransportException: [MSA-ES-CLUSTER_NODE_DATA2][inet[/10.250.132.219:9300]][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: [ubilogs-15.2.2][3]: query[filtered(+_id:AVMMO9EI4dV_htXhBe5Y)->cache(_type:logs)],from[0],size[120],sort[<custom:"date": org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource@1d087a3b>]: Query Failed [Failed to execute main query]
at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:163)
at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:301)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:312)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:776)
at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:767)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchException: org.elasticsearch.common.breaker.CircuitBreakingException: [FIELDDATA] Data too large, data for [date] would be larger than limit of [7710022041/7.1gb]
at org.elasticsearch.index.fielddata.plain.AbstractIndexFieldData.load(AbstractIndexFieldData.java:80)
at org.elasticsearch.index.fielddata.fieldcomparator.LongValuesComparatorSource$1.getLongValues(LongValuesComparatorSource.java:67)
at org.apache.lucene.search.FieldComparator$LongComparator.setNextReader(FieldComparator.java:716)
at
... 9 more

any idea on how to recover from this?

Antoine

2

Your fielddata cache has grown too big. See https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html#fielddata-size.

3

thank you.

to solve our issue we had to delete some documents from the index.
This is only a short term solution, we are also planning to add mode data node.
Currently we have 2 search nodes, 1 index node and 2 data nodes but with over 1.3 billions of documents the search queries were raising this circuit breaker exception when we were trying to sort by date. Without the sort it was OK

Antoine