Hello,
We just had something very weird on our ES cluster, and I wanted to know if
this behavior was either documented or expected or...
ES version: 1.0.1
1 node:
master: yes
http: yes
data: no
3 nodes:
master: no
http: yes
data: yes
The virtual machine with the master node was destroyed, so we had to
replace it with a brand new, bare one.
When the new master joined the cluster, the cluster went red, all shards
were marked as unassigned and started to initialize. Also all application
queries we sent to the cluster failed with:
[2014-11-12 21:32:00,120][DEBUG][action.search.type ] [some host]
[cluster][23], node[4ZYs5T2QSVG0pSRiE7zJDg], [R], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@32b3ab46]
org.elasticsearch.transport.RemoteTransportException: [some
host][inet[/1.2.3.4:9300]][search/phase/query+fetch]
Caused by: org.elasticsearch.ElasticsearchException:
java.lang.NumberFormatException: Invalid shift value (64) in prefixCoded
bytes (is encoded value really an INT?)
at
org.elasticsearch.index.fielddata.AbstractIndexFieldData.load(AbstractIndexFieldData.java:75)
at
org.elasticsearch.search.aggregations.support.FieldDataSource$MetaData.load(FieldDataSource.java:88)
at
org.elasticsearch.search.aggregations.support.AggregationContext.bytesField(AggregationContext.java:175)
at
org.elasticsearch.search.aggregations.support.AggregationContext.valuesSource(AggregationContext.java:126)
at
org.elasticsearch.search.aggregations.support.ValueSourceAggregatorFactory.create(ValueSourceAggregatorFactory.java:57)
at
org.elasticsearch.search.aggregations.AggregatorFactories.createSubAggregators(AggregatorFactories.java:55)
at
org.elasticsearch.search.aggregations.Aggregator.(Aggregator.java:78)
at
org.elasticsearch.search.aggregations.bucket.BucketsAggregator.(BucketsAggregator.java:46)
at
org.elasticsearch.search.aggregations.bucket.SingleBucketAggregator.(SingleBucketAggregator.java:32)
at
org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator.(FilterAggregator.java:50)
at
org.elasticsearch.search.aggregations.bucket.filter.FilterAggregator$Factory.create(FilterAggregator.java:91)
at
org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:127)
at
org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:79)
at
org.elasticsearch.search.query.QueryPhase.preProcess(QueryPhase.java:91)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:472)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:316)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:686)
at
org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryFetchTransportHandler.messageReceived(SearchServiceTransportAction.java:675)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2014-11-12 21:32:00,121][DEBUG][action.search.type ] [some host] All
shards failed for phase: [query_fetch]
Unfortunately, I was unable to reproduce it on another cluster.
So here are my guesses:
- There might have been some transient information I've lost with the
master node - Never have less than 2 master nodes, even temporarily.
- We have incidentally ran into mapping type gotchas as explained here.
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/mapping.html
I'm still bothered by the reshard and lose your mapping thing as I could
not reproduce it.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2dc96928-04bf-4b18-b599-b33e6609ae1c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.