Kibana Views Missing Data

[MrShiny]

Hi,

I've been using Kibana to visualise performance logs, for the last couple of months when visualising data is sometimes missing, as if during aggregation the index doesn't provide data. Normally a reload will fix this, but it's becoming more frequent, often becoming difficult to get a dashboard with complete data.

Things I've noticed:

• It's always an entire index which is affected (I index my logs by day, it's always intervals from 00:00 to 23:59, sometimes multi day)
• Sometimes an index provides some data, and across the entire time range (in the attached image, the top left graph shows two significant drops, but not zero data - these disappear on later reloads)
• It's a per-aggregation issue (the top left graph again, all that data is from the same index yet blue and yellow lines are unaffected

In the image:

• Top right and bottom left graphs are unaffected.
• Top left image has two day longdropouts
• Bottom right has two single day droupouts (one on red and one on blue) and a two day dropout (on red)
  
  

[spalger (Spencer Alger)]

Is this happening in non-timelion visualizations too?

[MrShiny]

I only have two non-timelion time-based graphs, I've never noticed this issue on them. The other graphs are pie charts or single-numbers so very difficult to tell.

[MrShiny]

I've just had the following error thrown for the top left graph (I assume, as it's the only one not rendered):

Error:  in cell #1: [reduce_search_phase_exception] [reduce] 
    at throwWithCell (/kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:30:11)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:156:13
    at arrayEach (/kibana/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (/kibana/node_modules/lodash/index.js:3345:13)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:148:9
    at bound (domain.js:280:14)
    at runBound (domain.js:293:12)
    at tryCatcher (/kibana/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/kibana/node_modules/bluebird/js/main/promise.js:503:31)
    at Promise._settlePromiseAt (/kibana/node_modules/bluebird/js/main/promise.js:577:18)
    at Promise._settlePromises (/kibana/node_modules/bluebird/js/main/promise.js:693:14)
    at Async._drainQueue (/kibana/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/kibana/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/kibana/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

[MrShiny]

Any update on this? Thanks.

[MrShiny]

Observed this in the kibana timelion window

[MrShiny]

and now this:

Timelion: Error: in cell #1: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@5efc8eeb on EsThreadPoolExecutor[search, queue capacity = 1000, …


    at throwWithCell (/kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:30:11)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:156:13
    at arrayEach (/kibana/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (/kibana/node_modules/lodash/index.js:3345:13)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:148:9
    at bound (domain.js:280:14)
    at runBound (domain.js:293:12)
    at tryCatcher (/kibana/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/kibana/node_modules/bluebird/js/main/promise.js:503:31)
    at Promise._settlePromiseAt (/kibana/node_modules/bluebird/js/main/promise.js:577:18)
    at Promise._settlePromises (/kibana/node_modules/bluebird/js/main/promise.js:693:14)
    at Async._drainQueue (/kibana/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/kibana/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/kibana/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

Which was prefixed by 151 copies of :

Error: in cell #1: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@5efc8eeb on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4735136b[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 44814176]] (and) [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@4969a57 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4735136b[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 44814184]] (and) ...