Kibana goes yellow state repeatedly

Elastic Stack Kibana
1

Hello, dear community,

We are having timeout and internal server errors but I couldn't find a way to fix it yet. I actually don't know where to look about these errors and I think because of these errors Kibana goes yellow state repeatedly. Let me give you some information about our cluster and the error.

In our cluster, there are 5 regions and 1 RTP machine.
I'm having this error only in 1 region and the RTP machine. The RTP is connected to the other regions via Remote Cluster. I have the error in NA Kibana and the RTP Kibana. The errors are identical.

{
  "cluster_name" : "na-production",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 9,
  "number_of_data_nodes" : 6,
  "active_primary_shards" : 910,
  "active_shards" : 1820,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

We have 3 master nodes:

  • Master-1 8GB RAM (A few data traffic)
  • Master-2 8GB RAM (A few data traffic)
  • Master-3 8GB RAM (A few data traffic)

We have 6 data nodes:

  • Hot-1 32GB RAM
  • Hot-2 32GB RAM
  • Warm-1 32GB RAM
  • Warm-2 32GB RAM
  • Cold-1 16GB RAM
  • Cold-2 16GB RAM

Cluster versions:

  • Elasticsearch: 7.15.2
  • Kibana: 7.15.2
  • Logstash: 7.15.2

The error log I have repeatedly on Kibana:

{"type":"log","@timestamp":"2022-02-23T03:18:08-05:00","tags":["error","plugins","data","data","indexPatterns"],"pid":13464,"message":"TimeoutError: Request timed out
    at ClientRequest.onTimeout (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/Connection.js:110:16)
    at ClientRequest.emit (events.js:400:28)\n    at Socket.emitRequestTimeout (_http_client.js:790:9)
    at Object.onceWrapper (events.js:519:28)
    at Socket.emit (events.js:412:35)
    at Socket._onTimeout (net.js:495:8)
    at listOnTimeout (internal/timers.js:557:17)
    at processTimers (internal/timers.js:500:7) {
  meta: {
    body: null,
    statusCode: null,
    headers: null,
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 3,
      aborted: false
    }
  },
  isBoom: true,
  isServer: true,
  data: null,
  output: {
    statusCode: 500,
    payload: {
      statusCode: 500,
      error: 'Internal Server Error',
      message: 'An internal server error occurred'
    },
    headers: {}
  }
}"}

I'm seeing this log all the time on Kibana and I couldn't find anything about it and I don't know where to check. Please help me understand this error, I don't want our Kibana to go yellow anymore..

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.