Kibana Views Missing Data

Elastic Stack Kibana
1

Hi,

I've been using Kibana to visualise performance logs, for the last couple of months when visualising data is sometimes missing, as if during aggregation the index doesn't provide data. Normally a reload will fix this, but it's becoming more frequent, often becoming difficult to get a dashboard with complete data.

Things I've noticed:

  • It's always an entire index which is affected (I index my logs by day, it's always intervals from 00:00 to 23:59, sometimes multi day)
  • Sometimes an index provides some data, and across the entire time range (in the attached image, the top left graph shows two significant drops, but not zero data - these disappear on later reloads)
  • It's a per-aggregation issue (the top left graph again, all that data is from the same index yet blue and yellow lines are unaffected

In the image:

  • Top right and bottom left graphs are unaffected.
  • Top left image has two day longdropouts
  • Bottom right has two single day droupouts (one on red and one on blue) and a two day dropout (on red)
    graphs.png1633×607 68.1 KB
2

Is this happening in non-timelion visualizations too?

3

I only have two non-timelion time-based graphs, I've never noticed this issue on them. The other graphs are pie charts or single-numbers so very difficult to tell.

4

I've just had the following error thrown for the top left graph (I assume, as it's the only one not rendered):

Error:  in cell #1: [reduce_search_phase_exception] [reduce] 
    at throwWithCell (/kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:30:11)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:156:13
    at arrayEach (/kibana/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (/kibana/node_modules/lodash/index.js:3345:13)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:148:9
    at bound (domain.js:280:14)
    at runBound (domain.js:293:12)
    at tryCatcher (/kibana/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/kibana/node_modules/bluebird/js/main/promise.js:503:31)
    at Promise._settlePromiseAt (/kibana/node_modules/bluebird/js/main/promise.js:577:18)
    at Promise._settlePromises (/kibana/node_modules/bluebird/js/main/promise.js:693:14)
    at Async._drainQueue (/kibana/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/kibana/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/kibana/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)
5

Any update on this? Thanks.

6

Observed this in the kibana timelion window

7

and now this:

Timelion: Error: in cell #1: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@5efc8eeb on EsThreadPoolExecutor[search, queue capacity = 1000, …


    at throwWithCell (/kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:30:11)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:156:13
    at arrayEach (/kibana/node_modules/lodash/index.js:1289:13)
    at Function.<anonymous> (/kibana/node_modules/lodash/index.js:3345:13)
    at /kibana/src/core_plugins/timelion/server/handlers/chain_runner.js:148:9
    at bound (domain.js:280:14)
    at runBound (domain.js:293:12)
    at tryCatcher (/kibana/node_modules/bluebird/js/main/util.js:26:23)
    at Promise._settlePromiseFromHandler (/kibana/node_modules/bluebird/js/main/promise.js:503:31)
    at Promise._settlePromiseAt (/kibana/node_modules/bluebird/js/main/promise.js:577:18)
    at Promise._settlePromises (/kibana/node_modules/bluebird/js/main/promise.js:693:14)
    at Async._drainQueue (/kibana/node_modules/bluebird/js/main/async.js:123:16)
    at Async._drainQueues (/kibana/node_modules/bluebird/js/main/async.js:133:10)
    at Immediate.Async.drainQueues (/kibana/node_modules/bluebird/js/main/async.js:15:14)
    at runCallback (timers.js:637:20)
    at tryOnImmediate (timers.js:610:5)
    at processImmediate [as _immediateCallback] (timers.js:582:5)

Which was prefixed by 151 copies of :

Error: in cell #1: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@5efc8eeb on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4735136b[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 44814176]] (and) [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$6@4969a57 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@4735136b[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 44814184]] (and) ...

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.