Hey,
Backstory:
We're using ELK to sift through logs.
I copied (filesystem) the elastisearch data directory from a backup on a new server where I launched a container form the same image that created the data, which runs elasticsearch 1.7.1.
When querying elasticsearch manually everything seems to be fine:
$ curl -XGET 'http://localhost:9200/logstash-2015.08.31/_count?pretty' -d '{
"query": { "match_all" :{}}}'
{
"count" : 3186523,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
}
}
Replacing _count with _search in that query also works fine.
Now on to Kibana 4.1.2:
When I try to Discover logs I get the error Courier Fetch: 5 of 5 shards failed and Elasticsearch produces some errors:
[2015-09-30 13:31:03,812][DEBUG][action.search.type ] [Marc Spector] [636] Failed to execute fetch phase
java.lang.NullPointerException
at org.elasticsearch.search.highlight.HighlightPhase.hitExecute(HighlightPhase.java:87)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:194)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:516)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:449)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Full error log: http://ix.io/l6c
Now why am I posting this here under kibana and not under elasticsearch you might ask...
The thing is: Visualize (and Dashboard) continue to work and do show me the requested data without any ES errors....
Where would I start looking to solve this issue?
Thank you for your time,
Jeroen