I have been away from using the ELK stack for over a year and there seems to have been a lot of changes. I have a job to bulk process about 20k of files which I have managed to get ELK to ingest but now that I need to process the data I am running into trouble. This query below is kibana looking at logs in a one hour timeframe on october the 1st and trying to analyse unique visitors. I am doing this by clicking on the Visualise button for unique clients.
This has worked for me in the past on older versions of ELK on much beefier machines so I don't know if this is the cause of resource starvation. In the past I would get OOM messages when I tried doing something like this in the past but in this case I have narrowed my field to 5000 hits and it is struggling to visualise this. Google searching for "Field data loading is forbidden on client" has not helped me. Looking at this https://www.elastic.co/guide/en/elasticsearch/guide/current/fielddata-intro.html I thing maybe I have a starved system but to me it feels like permission issues somewhere.
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on client"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-2015.10.01","node":"z4VjEsw1QkS7sVReb3XWvQ","reason":{"type":"exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on client","caused_by":{"type":"unchecked_execution_exception","reason":"java.lang.IllegalStateException: Field data loading is forbidden on client","caused_by":{"type":"illegal_state_exception","reason":"Field data loading is forbidden on client"}}}}]}}
KbnError@http://192.168.1.5:5601/bundles/commons.bundle.js:58153:30
RequestFailure@http://192.168.1.5:5601/bundles/commons.bundle.js:58186:19
http://192.168.1.5:5601/bundles/kibana.bundle.js:78760:57
http://192.168.1.5:5601/bundles/commons.bundle.js:60967:28
http://192.168.1.5:5601/bundles/commons.bundle.js:60936:31
map@[native code]
map@http://192.168.1.5:5601/bundles/commons.bundle.js:60935:34
callResponseHandlers@http://192.168.1.5:5601/bundles/kibana.bundle.js:78732:26
http://192.168.1.5:5601/bundles/kibana.bundle.js:78239:37
processQueue@http://192.168.1.5:5601/bundles/commons.bundle.js:42339:31
http://192.168.1.5:5601/bundles/commons.bundle.js:42355:40
$eval@http://192.168.1.5:5601/bundles/commons.bundle.js:43583:29
$digest@http://192.168.1.5:5601/bundles/commons.bundle.js:43394:37
$apply@http://192.168.1.5:5601/bundles/commons.bundle.js:43691:32
done@http://192.168.1.5:5601/bundles/commons.bundle.js:38140:54
completeRequest@http://192.168.1.5:5601/bundles/commons.bundle.js:38338:16
requestLoaded@http://192.168.1.5:5601/bundles/commons.bundle.js:38279:25