Beta: Fielddata

DigiAngel · October 29, 2016, 2:01am

So my testing box locked up...had to hard power it down. Now I am seeing this:

[2016-10-28T19:56:19,264][DEBUG][o.e.a.s.TransportSearchAction] [scanner] All shards failed for phase: [query]
org.elasticsearch.transport.RemoteTransportException: [scanner][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.
	at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:335) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:111) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.config(ValuesSourceAggregationBuilder.java:355) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:305) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:299) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregatorFactory.<init>(ValuesSourceAggregatorFactory.java:40) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregatorFactory.<init>(DateHistogramAggregatorFactory.java:82) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.bucket.histogram.DateHistogramAggregationBuilder.innerBuild(DateHistogramAggregationBuilder.java:248) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:300) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:49) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:126) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:211) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:696) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:534) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:501) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:242) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$6(SearchTransportService.java:276) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.transport.TransportService$6.doRun(TransportService.java:548) [elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:504) [elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.0.0-beta1.jar:5.0.0-beta1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

also now almost all of my fields have a fieldname.keyword as well. Can someone shed some light on what this is? Thank you.

mvg · October 31, 2016, 10:58am

Elasticsearch 5.x will automatically upgrade string fields to either text or keyword field type. The string field has been removed since 5.0. The text field requires field data for any aggregation like operation. So you will need to enabled that. It is disabled by default, because field data on text field can require a lot of memory. So you have to be cautious when enabling field data on text fields.

I don't think the error has anything to do with the fact that your restarted ES. This error just happens when searches are being executed that have aggregations on a text field.

DigiAngel · October 31, 2016, 3:03pm

Thanks Martijn. Wow did I hose explaining this....sorry about that Ok so first off, after removing all indexes, mappings, and templates...basically restarting, here's what the mappings look like:

there's no .keywoard, and, per my mapping, all stings are set to not analyzed. After doing a system upgrade (Ubuntu 16.04 server) (last log here):

Start-Date: 2016-10-28  08:48:14
Commandline: apt-get dist-upgrade
Requested-By: (1000)
Upgrade: libc6-dev:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), libmysqlclient-dev:amd64 (5.7.15-0ubuntu0.16.04.1, 5.7.16-0ubuntu0.16.04.1), libc6:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), locales:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), language-pack-en:amd64 (1:16.04+20160627, 1:16.04+20161009), libapparmor1:amd64 (2.10.95-0ubuntu2.2, 2.10.95-0ubuntu2.5), libapparmor-perl:amd64 (2.10.95-0ubuntu2.2, 2.10.95-0ubuntu2.5), python3-distupgrade:amd64 (1:16.04.16, 1:16.04.17), libc-bin:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), ubuntu-release-upgrader-core:amd64 (1:16.04.16, 1:16.04.17), mysql-common:amd64 (5.7.15-0ubuntu0.16.04.1, 5.7.16-0ubuntu0.16.04.1), libmysqlclient20:amd64 (5.7.15-0ubuntu0.16.04.1, 5.7.16-0ubuntu0.16.04.1), libc-dev-bin:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), multiarch-support:amd64 (2.23-0ubuntu3, 2.23-0ubuntu4), apparmor:amd64 (2.10.95-0ubuntu2.2, 2.10.95-0ubuntu2.5)

the machine had an issue...what the issue was I don't know, but I got no response from it during this upgrade. Once getting physical access and restarting is when I saw that all fields were checkmarked analyzed, and each string had a secondary, for example, curve had curve.keyword. So if that's expected behavior, why don't I see them now? And how do I enable the text field item? Thank you....hope that makes it clearer.

DigiAngel · October 31, 2016, 5:40pm

Ok so I'm doing more reading here: Field Data but I'm not sure exactly what I need to do, if anything, on my end. I don't see this right now...but there are several visualizations that rely on a field called "services". When this change happened I had to switch them all to "services.keyword". Thanks again.

DigiAngel · November 3, 2016, 4:40pm

Circling back on this....after running 2 days the ".keyword" popped up again, so this looks like normal behavior. This go round however my string fields remained non-analyzed. I did have to change all my visualizations to point to the .keyword fields afterwards. Is there a way to force this on install or upgrade instead of having it happening randomly? Thank you.