We are having performance issue in Elasticsearch/Kibana after migrating to another server host

Elastic Stack Kibana
1

Hello,
We are having issue in Kibana.
We have AWS Proxy and Kibana 7.1 setup in the same server. After migrating both to another server host and make it live in the load balancer, we are facing request timeout issue when accessing our Kibana application.
image

Though when using the old server back in the load balancer, it works perfectly fine.
Does anyone know why this is happening? Any related information will be appreciated. Thank you!

PS: Both servers has the same specification.

By the way, here are the settings that we have:
AWS Elasticsearch Proxy
aws-es-proxy-0.9-windows-amd64.exe -endpoint "https://vpc_endpoint " -listen "127.0.0.1:9230"

Kibana.yml

server.port: 5609
server.host: "x.x.x.x"
server.basePath: "/7.1/kibana"
elasticsearch.hosts: "http://127.0.0.1:9230/"
elasticsearch.ssl.verificationMode: none
elasticsearch.requestTimeout: 300000

2

Anyone could help on this issue?

3

Kibana is timing out communicating with Elasticsearch. What is between Kibana and Elasticsearch? Is there also a proxy there?

4

We only used proxy in Elasticsearch.
We defined elasticsarch.hosts in kibana.yml to connect our Kibana to Elasticsarch,

Does it ES having the main cause here?

5

To confirm, is elasticsearch.hosts pointing to a proxy, or an Elasticsearch node?

If it's a proxy, can you try changing it to point directly to an Elasticsearch node to rule out the issue being the proxy?

Other than being network related, it could be that Elasticsearch is far too overwhelmed to serve any requests. What is the health of the cluster at this time? Does the Cluster Health ever go red during this time?

6

Hi @tylersmalley,

Can you provide an Elasticsearch node sample?
I believe elasticsearch.hosts is pointing to a proxy.

We are currently pointing on the server host that doesn't have the issue.
Here is the status of the Cluster Health right now.

{
"cluster_name" : "xxxxxx:xxx-xxxx-prod",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 23,
"number_of_data_nodes" : 20,
"discovered_master" : true,
"active_primary_shards" : 680,
"active_shards" : 1360,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

I'll try to point ES/Kibana in the server host having the said issue and check the Cluster Health Status.

Thanks for helping in this matter.

7

image
image1026×592 35.6 KB

Uploading here the status of the Cluster.

8

Could anyone help me with this? This is a production issue.

Here's the comparison of the instances where we're running Proxy and Kibana.

Old - which having no issue
Windows Server 2012 R2
Processor: Inter(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz 2.90 GHz
Installed memory (RAM): 30.0 GB
System type: 64-bit OS, x64-based processor
Virtual processor: 16

New - which has the issue
Windows Server 2016
Processor: Inter(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz 3.00 GHz
Installed memory (RAM): 31.0 GB
System type: 64-bit OS, x64-based processor
Virtual processor: 8

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.